Zulip Documentation Release 1.4.0
The Zulip Team
Feb 07, 2017
Overview
1 Zulip overview 3 1.1 Community...... 3 1.2 Installing the Zulip Development environment...... 4 1.3 Running Zulip in production...... 4 1.4 Ways to contribute...... 4 1.5 Google Summer of Code...... 4 1.6 How to get involved with contributing to Zulip...... 4 1.7 License...... 5
2 Zulip architectural overview 7 2.1 Key Codebases...... 7 2.2 Usage assumptions and concepts...... 7 2.3 Components...... 8 2.4 Glossary...... 11
3 Directory structure 13 3.1 Core Python files...... 13 3.2 HTML Templates...... 13 3.3 JavaScript and other static assets...... 14 3.4 Tests...... 14 3.5 Management commands...... 14 3.6 Scripts...... 14 3.7 API and Bots...... 15 3.8 Production puppet configuration...... 15 3.9 Additional Django apps...... 15 3.10 Jinja2 Compatibility Files...... 15 3.11 Translation files...... 16 3.12 Documentation...... 16
4 Zulip Roadmap 17 4.1 Introduction...... 17 4.2 Major projects...... 17 4.3 Core User Experience...... 18 4.4 Social features...... 18 4.5 Real-time sync...... 18 4.6 Onboarding issues...... 18 4.7 Production installation issues...... 19
i 4.8 Administration and management...... 19 4.9 Scalability and performance...... 19 4.10 Technology improvements...... 19 4.11 Technical Debt...... 20 4.12 Security...... 20 4.13 Testing...... 20 4.14 Documentation...... 20 4.15 Integrations and bots...... 20 4.16 Android app...... 21 4.17 iOS app...... 21 4.18 Server/webapp support for mobile...... 21 4.19 Desktop apps...... 21 4.20 Community...... 21
5 Version History 23 5.1 Unreleased...... 23 5.2 1.5.0 – 2017-02-06...... 23 5.3 1.4.3 - 2017-01-29...... 26 5.4 1.4.2 - 2016-09-27...... 26 5.5 1.4.1 - 2016-09-03...... 26 5.6 1.4.0 - 2016-08-25...... 26 5.7 1.3.13 - 2016-06-21...... 29 5.8 1.3.12 - 2016-05-10...... 30 5.9 1.3.11 - 2016-05-02...... 30 5.10 1.3.10 - 2016-01-21...... 31 5.11 1.3.9 - 2015-11-16...... 31 5.12 1.3.8 - 2015-11-15...... 31 5.13 1.3.7 - 2015-10-19...... 32
6 Requirements 33 6.1 Server...... 33 6.2 Credentials needed...... 34
7 Production Installation 35 7.1 Step 0: Subscribe...... 35 7.2 Step 1: Install SSL Certificates...... 35 7.3 Step 2: Download and install latest release...... 35 7.4 Step 3: Configure Zulip...... 36 7.5 Step 4: Initialize Zulip database...... 36 7.6 Step 5: Create a Zulip organization and login...... 36 7.7 Troubleshooting...... 37
8 Troubleshooting 39 8.1 Using supervisorctl...... 39 8.2 Troubleshooting services...... 41
9 Customize Zulip 43 9.1 Integrations...... 43 9.2 Streams and Topics...... 44 9.3 Notification settings...... 44 9.4 Mobile and desktop apps...... 44 9.5 All other features...... 44 9.6 Enjoy your Zulip installation!...... 44
10 Secure, maintain, and upgrade 47 ii 10.1 Upgrading...... 47 10.2 Upgrading from a git repository...... 49 10.3 Backups...... 49 10.4 Monitoring...... 51 10.5 Scalability...... 51 10.6 Securing your Zulip server...... 52 10.7 Management commands...... 52
11 Security Model 55 11.1 Secure your Zulip server like your email server...... 55 11.2 Encryption and Authentication...... 55 11.3 Messages and History...... 56 11.4 Users and Bots...... 57 11.5 User-uploaded content...... 57 11.6 Final notes and security response...... 58
12 Authentication methods 59 12.1 Adding additional methods using python-social-auth...... 59 12.2 Remote User SSO Authentication...... 60
13 Postgres database details 63 13.1 Remote Postgres database...... 63 13.2 Debugging postgres database issues...... 64 13.3 Stopping the Zulip postgres database...... 64 13.4 Debugging issues starting postgres...... 64 13.5 Postgres Vacuuming alerts...... 65
14 Development environment installation 67 14.1 Requirements...... 67 14.2 Recommended setup (Vagrant)...... 67 14.3 Advanced setup (non-Vagrant)...... 67 14.4 Slow internet connections...... 68 14.5 Installing remotely...... 68 14.6 Next steps...... 68
15 Vagrant environment setup tutorial 69 15.1 Requirements...... 70 15.2 Step 1: Install Prerequisites...... 70 15.3 Step 2: Get Zulip Code...... 72 15.4 Step 3: Start the development environment...... 73 15.5 Step 4: Developing...... 75 15.6 Next Steps...... 77 15.7 Troubleshooting & Common Errors...... 77 15.8 Specifying a proxy...... 82
16 Zulip development environment setup without Vagrant 83 16.1 Installing directly on Ubuntu...... 83 16.2 Installing manually on Linux...... 83 16.3 Using Docker (experimental)...... 89
17 Using the Development Environment 91
18 Developing on a remote machine 93 18.1 Connecting to the remote environment...... 93 18.2 Setting up the development environment...... 93
iii 18.3 Running the development server...... 94 18.4 Making changes to code on your remote development server...... 94
19 Writing a new integration 99 19.1 Types of integrations...... 99 19.2 General advice...... 100 19.3 Webhook integrations...... 100 19.4 Python script and plugin integrations...... 101 19.5 Documenting your integration...... 102 19.6 Hello World webhook Walkthrough...... 102
20 Writing a new application feature 109 20.1 General Process in brief...... 109 20.2 Example Feature...... 110
21 Writing views in Zulip 115 21.1 What this covers...... 115 21.2 What is a view?...... 115 21.3 Modifying urls.py...... 115 21.4 Writing human-readable views...... 116 21.5 Writing API REST endpoints...... 116 21.6 Legacy endpoints used by the web client...... 120 21.7 Webhook integration endpoints...... 120
22 Life of a Request 121 22.1 A request is sent to the server, and handled by Nginx...... 121 22.2 Nginx secures traffic with SSL...... 121 22.3 Static files are served directly by Nginx...... 121 22.4 Nginx routes other requests between tornado and django...... 122 22.5 Django routes the request to a view in urls.py files...... 122 22.6 Views serving HTML are internationalized by server path...... 122 22.7 API endpoints use REST...... 122 22.8 Django calls rest_dispatch for REST endpoints, and authenticates...... 123 22.9 The view will authorize the user, extract request variables, and validate them...... 124 22.10 Results are given as JSON...... 124
23 Reading list 125 23.1 General programming/IT...... 125 23.2 Python...... 126 23.3 Java/Android...... 126 23.4 JavaScript/ECMAScript...... 126 23.5 Git/Version Control Systems (VCS)...... 126 23.6 Computer Science/Algorithms...... 126 23.7 Community experience...... 127 23.8 Competitions/Camps...... 127 23.9 Massive Open Online Courses (MOOC) Platforms...... 127
24 Git & GitHub Guide 129 24.1 Quick start: How Zulip uses Git and GitHub...... 129 24.2 Set up Git...... 130 24.3 How Git is different...... 130 24.4 Important Git terms...... 131 24.5 Get Zulip code...... 133 24.6 Using Git as you work...... 134 24.7 Create a pull request...... 141 iv 24.8 Update a pull request...... 143 24.9 Collaborate...... 143 24.10 Review changes...... 144 24.11 Get and stay out of trouble...... 144 24.12 Zulip-specific tools...... 148
25 Version control 151 25.1 Commit Discipline...... 151 25.2 Commit Messages...... 152
26 Code style and conventions 155 26.1 Be consistent!...... 155 26.2 Lint tools...... 155 26.3 Secrets...... 156 26.4 Dangerous constructs...... 156 26.5 JS array/object manipulation...... 158 26.6 More arbitrary style things...... 158
27 Python static type checker (mypy) 161 27.1 type_debug.py ...... 161 27.2 Zulip goals...... 162 27.3 Installing mypy...... 162 27.4 Running mypy on Zulip’s code locally...... 162 27.5 Excluded files...... 163 27.6 Mypy is there to find bugs in Zulip before they impact users...... 163 27.7 Annotating strings...... 163
28 Reviewing Zulip server code 165 28.1 Things to look for...... 165 28.2 Tooling...... 166 28.3 Additional Resources...... 166
29 Testing and writing tests 167 29.1 Overview...... 167 29.2 Running tests...... 167 29.3 Schema and initial data changes...... 168 29.4 Wiping the test databases...... 168 29.5 Local browser testing (local app + web browser)...... 169 29.6 Python 3 Compatibility...... 169
30 Linters 171 30.1 Overview...... 171 30.2 Running the linters...... 171 30.3 General considerations...... 172 30.4 Lint checks...... 172 30.5 lint-all...... 173 30.6 Philosophy...... 174
31 JavaScript unit tests 177 31.1 HTML output...... 177 31.2 Coverage reports...... 178 31.3 Handling dependencies in unit tests...... 178 31.4 Creating new test modules...... 179
32 Backend Django tests 181
v 32.1 Overview...... 181 32.2 Running tests...... 181 32.3 How to write tests...... 182 32.4 Zulip Testing Philosophy...... 182 32.5 Testing considerations...... 184
33 Web frontend black-box casperjs tests 187 33.1 Debugging Casper.JS...... 187 33.2 Writing Casper tests...... 188
34 Manual testing 191 34.1 Basic Stuff...... 191
35 Settings system 203 35.1 Server settings...... 203 35.2 Realm settings...... 204
36 Queue processors 207 36.1 Adding a new queue processor...... 207 36.2 Publishing events into a queue...... 208 36.3 Clearing a RabbitMQ queue...... 208
37 Writing bots 209 37.1 How to deploy a bot...... 209 37.2 How to develop a bot...... 210 37.3 Bot API...... 211 37.4 Common problems...... 214
38 Custom Apps 215 38.1 Definition...... 215 38.2 Problem statement...... 215 38.3 A quick note on bots/integrations...... 215 38.4 Categories of custom apps...... 216 38.5 World Reader...... 217 38.6 Zulip Reader...... 219 38.7 Deployment issues...... 219
39 Unread counts and the pointer 223 39.1 Pointer logic...... 223 39.2 Unread count logic...... 224
40 Markdown implementation 225 40.1 Testing...... 225 40.2 Changing Zulip’s markdown processor...... 225 40.3 Zulip’s Markdown philosophy...... 226 40.4 Zulip’s Changes to Markdown...... 227
41 Realms in Zulip 229 41.1 Creating Realms...... 229 41.2 Subdomains...... 230
42 Static asset pipeline 231 42.1 Primary build process...... 231 42.2 Webpack/CommonJS modules...... 231 42.3 Adding static files...... 232
vi 43 Schema Migrations 233
44 HTML and CSS 235 44.1 Zulip CSS organization...... 235 44.2 Editing Zulip CSS...... 235 44.3 CSS Style guidelines...... 236 44.4 Validating CSS...... 236
45 Emoji 237 45.1 Emoji codes...... 237 45.2 Tooling...... 237
46 Full-text search 239 46.1 The default full-text search implementation...... 239 46.2 An optional full-text search implementation...... 239
47 Translating Zulip 241 47.1 Translation style guides...... 241 47.2 Translation process...... 241 47.3 Translators’ workflow...... 242 47.4 Testing translations...... 242 47.5 Setting the default language in Zulip...... 243 47.6 Translation resource files...... 243 47.7 Backend translations...... 243 47.8 Frontend translations...... 244 47.9 Transifex config...... 245
48 Transifex CLI setup 247
49 Logging and Performance Debugging 249
50 Zulip server release checklist 251 50.1 A week before the release...... 251 50.2 Final release preparation...... 251 50.3 Executing the release...... 251
51 Documentation 253 51.1 Developer and sysadmin documentation...... 253 51.2 Core website documentation...... 254 51.3 General user documentation...... 254
52 General user guide documentation 255 52.1 Editing and testing...... 255 52.2 Writing documentation...... 255 52.3 Features...... 257 52.4 Documentation template...... 264
vii viii Zulip Documentation, Release 1.4.0
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Our code is available at our GitHub repository. This set of documents covers installation and contribution instructions. Contents: • Overview • Zulip in production • Development environment • Developer tutorials • Code contribution guide • Code testing • Subsystem documentation Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved | License
Overview 1 Zulip Documentation, Release 1.4.0
2 Overview CHAPTER 1
Zulip overview
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org.
1.1 Community
There are several places online where folks discuss Zulip. One of those places is our public Zulip instance. You can go through the simple signup process at that link, and then you will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luck finding core developers roughly between 16:00 UTC and 23:59 UTC. Most questions get a reply within minutes to a few hours, depending on time of day. For Google Summer of Code students and applicants, we have a mailing list for help, questions, and announcements. We have a public mailing list that is currently pretty low traffic because most discussions happen in our public Zulip instance. We use it to announce Zulip developer community gatherings and ask for feedback on major technical or design decisions. It has several hundred subscribers, so you can use it to ask questions about features or possible bugs, but please don’t use it ask for generic help getting started as a contributor (e.g. because you want to do Google Summer of Code). The rest of this page covers how to get involved in the Zulip project in detail. Zulip also has a blog. Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a GitHub account should be able to create Issues there pertaining to bugs or enhancement requests. We also use Pull Requests as our primary mechanism to receive code contributions. The Zulip community has a Code of Conduct.
3 Zulip Documentation, Release 1.4.0
1.2 Installing the Zulip Development environment
The Zulip development environment is the recommended option for folks interested in trying out Zulip. This is documented in the developer installation guide.
1.3 Running Zulip in production
Zulip in production supports Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial. Work is ongoing on adding support for additional platforms. The installation process is documented at https://zulip.org/server.html and in more detail in the documentation.
1.4 Ways to contribute
Zulip welcomes all forms of contributions! The page documents the Zulip development process. • Pull requests. Before a pull request can be merged, you need to sign the Dropbox Contributor License Agree- ment. Also, please skim our commit message style guidelines. • Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also run them all in your development environment following the instructions in the testing docs. You can also try out our new desktop client, which is in alpha; we’d appreciate testing and feedback. • Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs. Recommended reading for new contributors includes the directory structure and new feature tutorial. You can also improve Zulip.org. • Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . There are also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas where you’d love to see more documentation to the relevant list! Please report any security issues you discover to [email protected]. • App codebases. This repository is for the Zulip server and web app (including most integrations); the desktop, Android, and iOS apps, are separate repositories, as are our experimental React Native iOS app and alpha Electron desktop app. • Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine, and Trello), plus node.js API bindings, an isomorphic JavaScript library, and a full-text search PostgreSQL extension, as separate repos. • Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to our translations. See our translating documentation if you’re interested in contributing!
1.5 Google Summer of Code
We participated in GSoC last year and hope to do so again in 2017. For guidance, please read our GSoC instructions and ideas page and feel free to email our GSoC mailing list.
1.6 How to get involved with contributing to Zulip
First, subscribe to the Zulip development discussion mailing list.
4 Chapter 1. Zulip overview Zulip Documentation, Release 1.4.0
The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have your own project idea in mind or want to get some experience with working on Zulip before embarking on a larger project you have in mind: • Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writing an integration can be a great first contribution. There’s detailed documentation on how to write integrations in the Zulip integration writing guide. • Bite Size: Smaller projects that might be a great first contribution. • Documentation: The Zulip project loves contributions of new documentation. • Help Wanted: A broader list of projects that nobody is currently working on. • Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms. • Bugs: Open bugs. • Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip users are excited about. • 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’re looking to make an impact. Another way to find issues in Zulip is to take advantage of our “area:” convention in separating out issues. We partition all of our issues into areas like admin, compose, emoji, hotkeys, i18n, onboarding, search, etc. You can see this here: [https://github.com/zulip/zulip/labels] Click on any of the “area:” labels and you will see all the tickets related to your area of interest. If you’re excited about helping with an open issue, just post on the conversation thread that you’re working on it. You’re encouraged to ask questions on how to best implement or debug your changes – the Zulip maintainers are excited to answer questions to help you stay unblocked and working efficiently. We also welcome suggestions of features that you feel would be valuable or changes that you feel would make Zulip a better open source project, and are happy to support you in adding new features or other user experience improvements to Zulip. If you have a new feature you’d like to add, we recommend you start by opening a GitHub issue about the feature idea explaining the problem that you’re hoping to solve and that you’re excited to work on it. A Zulip maintainer will usually reply within a day with feedback on the idea, notes on any important issues or concerns, and and often tips on how to implement or test it. Please feel free to ping the thread if you don’t hear a response from the maintainers – we try to be very responsive so this usually means we missed your message. For significant changes to the visual design, user experience, data model, or architecture, we highly recommend posting a mockup, screenshot, or description of what you have in mind to zulip-devel@ to get broad feedback before you spend too much time on implementation details. Finally, before implementing a larger feature, we highly recommend looking at the new feature tutorial and coding style guidelines on ReadTheDocs. Feedback on how to make this development process more efficient, fun, and friendly to new contributors is very welcome! Just send an email to the Zulip Developers list with your thoughts.
1.7 License
Copyright 2011-2016 Dropbox, Inc. and contributors Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
1.7. License 5 Zulip Documentation, Release 1.4.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. The software includes some works released by third parties under other free and open source licenses. Those works are redistributed under the license terms under which the works were received. For more details, see the docs/ THIRDPARTY file included with this distribution.
6 Chapter 1. Zulip overview CHAPTER 2
Zulip architectural overview
2.1 Key Codebases
The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon to also support Python 3) and using the Django framework. That codebase includes server-side code and the web client, as well as Python API bindings and most of our integrations with other services and applications (see the directory structure guide). We maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations with Phabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL exten- sion. Our mobile clients are separate code repositories: Android, iOS (stable), and our experimental React Native iOS app. Our legacy desktop application (implemented in QT/WebKit) and our new, alpha cross-platform desktop app (implemented in Electron) are also separate repositories. We use Transifex to do translations. In this overview, we’ll mainly discuss the core Zulip server and web application.
2.2 Usage assumptions and concepts
Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a small team to more than a thousand users. It features real-time notifications, message persistence and search, public group conversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, team presence/buddy lists, a rich API, Markdown message support, and numerous integrations with other services. The maintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and Mac OS X clients, as well as people using modern web browsers or dedicated Zulip API clients. A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber with its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip realms. The administrators of a realm can choose whether to allow anyone to register an account and join, or only allow people who have been invited, or restrict registrations to members of particular groups (using email domain
7 Zulip Documentation, Release 1.4.0 names or corporate single-sign-on login for verification). For more on security considerations, see the security model section. The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldest message that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). The home screen displays the most recent messages in all the streams a user has joined (except for the streams they’ve muted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow has its own URL. The user can quickly see what conversation they’re in – the stream and topic, or the names of the user(s) they’re private messaging with – using the recipient bar displayed atop each conversation. Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming information flow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they find irrelevant.
2.3 Components
2.3.1 Tornado and Django
We use both the Tornado and Django Python web frameworks. Django is the main web application server; Tornado runs the server-to-client real-time push system. The app servers are configured by the Supervisor configuration (which explains how to start the server processes; see “Supervisor” below) and the nginx configuration (which explains which HTTP requests get sent to which app server). Tornado is an asynchronous server and is meant specifically to hold open tens of thousands of long-lived (long-polling or websocket) connections – that is to say, routes that maintain a persistent connection from every running client. For
8 Chapter 2. Zulip architectural overview Zulip Documentation, Release 1.4.0
this reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in Tornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip very much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those blocking requests carry a very high performance penalty for a single-threaded, asynchronous server. The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Django application server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on the code path for sending messages.
2.3.2 nginx
nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django and Tornado. It handles HTTP requests according to the rules laid down in the many config files found in zulip/puppet/zulip/ files/nginx/. zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of these files. It explains what happens when requests come in from outside. • In production, all requests to URLs beginning with /static/ are served from the corresponding files in / home/zulip/prod-static/, and the production build process (tools/build-release-tarball) compiles, minifies, and installs the static assets into the prod-static/ tree form. In development, files are served directly from /static/ in the git repository. • Requests to /json/events, /api/v1/events, and /sockjs are sent to the Tornado server. These are requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connection with Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server via puppet/zulip/files/nginx/zulip-include-frontend/upstreams. • Requests to all other paths are sent to the Django app via the UNIX socket unix:/home/ zulip/deployments/uwsgi-socket (defined in puppet/zulip/files/nginx/ zulip-include-frontend/upstreams). We use zproject/wsgi.py to implement uWSGI here (see django.core.wsgi).
2.3.3 Supervisor
We use supervisord to start server processes, restart them automatically if they crash, and direct logging. The config file is zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf. This is where Tor- nado and Django are set up, as well as a number of background processes that process event queues. We use event queues for the kinds of tasks that are best run in the background because they are expensive (in terms of performance) and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing guide.
2.3.4 memcached memcached is used to cache database model objects. zerver/lib/cache.py and zerver/lib/ cache_helpers.py manage putting things into memcached, and invalidating the cache when values change. The memcached configuration is in puppet/zulip/files/memcached.conf.
2.3.5 Redis
Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, a per-user rate limiting scheme example), and the email-to-Zulip integration.
2.3. Components 9 Zulip Documentation, Release 1.4.0
Redis is configured in zulip/puppet/zulip/files/redis and it’s a pretty standard configuration except for the last line, which turns off persistence:
# Zulip-specific configuration: disable saving to disk. save""
memcached was used first and then we added Redis specifically to implement rate limiting. We’re discussing switching everything over to Redis.
2.3.6 RabbitMQ
RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initial configuration happens in zulip/scripts/setup/configure-rabbitmq. We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, some analytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used for communication between the application server and the Tornado push system. Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/zerver/lib/queue.py. There’s an asynchronous client for use in Tornado and a more general client for use elsewhere. zerver/tornado/event_queue.py has helper functions for putting events into one queue or another. Most of the processes started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle them. Also see the queuing guide.
2.3.7 PostgreSQL
PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to live beyond a user’s current session. In production, Postgres is installed with a default configuration. The directory that would contain configuration files (puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Post- gresql extension. In a development environment, configuration of that postgresql extension is handled by tools/ postgres-init-dev-db (invoked by tools/provision). That file also manages setting up the development postgresql user. tools/provision also invokestools/do-destroy-rebuild-database‘ to create the actual database with its schema.
2.3.8 Nagios
Nagios is an optional component used for notifications to the system administrator, e.g., in case of outages. zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from puppet/zulip/files/ nagios_plugins/. This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the Zulip Nagios plugins are intended to be run on the Zulip servers themselves, and are included with the relevant component of the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under /usr/lib/ nagios/plugins/zulip_postgres_common).
10 Chapter 2. Zulip architectural overview Zulip Documentation, Release 1.4.0
2.4 Glossary
This section gives names for some of the elements in the Zulip UI used in Zulip development conversations. Contri- butions to extend this list are welcome! • chevron: A small downward-facing arrow next to a message’s timestamp, offering contextual options, e.g., “Reply”, “Mute [this topic]”, or “Link to this conversation”. To avoid visual clutter, the chevron only appears in the web UI upon hover. • message editing: If the realm admin allows it, then after a user posts a message, the user has a few minutes to click “Edit” and change the content of their message. If they do, Zulip adds a marker such as “(EDITED)” at the top of the message, visible to anyone who can see the message. • recipient bar: A visual indication of the context of a message or group of messages, displaying the stream and topic or private message recipient list, at the top of a group of messages. A typical 1-line message to a new recipient shows to the user as three lines of content: first the recipient bar, second the sender’s name and avatar alongside the timestamp (and, on hover, the star and the chevron), and third the message content. The recipient bar is or contains hyperlinks to help the user narrow. • star: Zulip allows a user to mark any message they can see, public or private, as “starred”. A user can easily ac- cess messages they’ve starred through the “Starred messages” link in the menu near “Home”, or use “is:starred” as a narrow or a search constraint. Whether a user has or has not starred a particular message is private; other users and realm admins don’t know whether a message has been starred, or by whom.
2.4. Glossary 11 Zulip Documentation, Release 1.4.0
12 Chapter 2. Zulip architectural overview CHAPTER 3
Directory structure
This page documents the Zulip directory structure, where to find things, and how to decide where to put a file. You may also find the new application feature tutorial helpful for understanding the flow through these files.
3.1 Core Python files
Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers. • zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions or templates. • zerver/models.py Main Django models file. Defines Zulip’s database tables. • zerver/lib/actions.py Most code doing writes to user-facing database tables. • zerver/views/*.py Most Django views. • zerver/webhooks/ Webhook views and tests for Zulip webhook integrations. • zerver/tornado/views.py Tornado views. • zerver/worker/queue_processors.py Queue workers. • zerver/lib/*.py Most library code. • zerver/lib/bugdown/ Backend Markdown processor. • zproject/backends.py Authentication backends.
3.2 HTML Templates
See our translating docs for details on Zulip’s templating systems.
13 Zulip Documentation, Release 1.4.0
• templates/zerver/ For Jinja2 templates for the backend (for zerver app). • static/templates/ Handlebars templates for the frontend.
3.3 JavaScript and other static assets
• static/js/ Zulip’s own JavaScript. • static/styles/ Zulip’s own CSS. • static/images/ Zulip’s images. • static/third/ Third-party JavaScript and CSS that has been vendored. • node_modules/ Third-party JavaScript installed via npm. • static/assets/ For assets not to be served to the web (e.g. the system to generate our favicons).
3.4 Tests
• zerver/tests/ Backend tests. • frontend_tests/node_tests/ Node Frontend unit tests. • frontend_tests/casper_tests/ Casper frontend tests. • tools/test-* Developer-facing test runner scripts.
3.5 Management commands
These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access). • zerver/management/commands/ Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly).
3.6 Scripts
• scripts/ Scripts that production deployments might run manually (e.g., restart-server). • scripts/lib/ Scripts that are needed on production deployments but humans should never run directly. • scripts/setup/ Scripts that production deployments will only run once, during installation. • tools/ Scripts used only in a Zulip development environment. These are not included in production release tarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production acciden- tally (e.g. things that delete the Zulip database without prompting).
14 Chapter 3. Directory structure Zulip Documentation, Release 1.4.0
• tools/setup/ Subdirectory of tools/ for things only used during the development environment setup process. • tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actual test suites should go in tools/.
3.7 API and Bots
• api/ Zulip’s Python API bindings (released separately). • api/examples/ API examples. • api/integrations/ Bots distributed as part of the Zulip API bundle. • bots/ Previously Zulip internal bots. These usually need a bit of work.
3.8 Production puppet configuration
This is used to deploy essentially all configuration in production. • puppet/zulip/ For configuration for production deployments. • puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.
3.9 Additional Django apps
• confirmation Email confirmation system. • analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). • corporate The old Zulip.com website. Not included in production distribution. • zilencer Primarily used to hold management commands that aren’t used in production. Not included in production distribution.
3.10 Jinja2 Compatibility Files
• zproject/jinja2/__init__.py Jinja2 environment. • zproject/jinja2/backends.py Jinja2 backend. • zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.
3.7. API and Bots 15 Zulip Documentation, Release 1.4.0
3.11 Translation files
• locale/ Backend (Django) translations data files. • static/locale/ Frontend translations data files.
3.12 Documentation
• docs/ Source for this documentation.
You can consult the repository’s .gitattributes file to see exactly which components are excluded from produc- tion releases (release tarballs are generated using tools/build-release-tarball).
16 Chapter 3. Directory structure CHAPTER 4
Zulip Roadmap
4.1 Introduction
Zulip has received a great deal of interest and attention since it was released as free and open source software by Dropbox. That attention has come with a lot of active development work from members of the Zulip community. From when Zulip was released as open source in late September 2015 through today (early November, 2016), more than 150 people have contributed over 1000 pull requests to the various Zulip repositories, the vast majority of which were submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges the pull requests). In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where the project is hoping to improve. This can be especially important in an open source project like Zulip, where development is distributed across many people around the world. This roadmap is intended to organize a list of the most important improvements that should be made to Zulip in the relatively near future. Our aim is to complete most of these improvements by February 2017 and then prepare a new roadmap then. This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will be used by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product and hold ourselves accountable for making Zulip improve rapidly. This roadmap is the best place for contributors to look for substantial projects that will definitely be of value to the community (if you’re looking for a starter project, see the guide to getting involved with Zulip). We periodically update this roadmap by adding strikethrough to issues that have been resolved, but the linked GitHub issues are the most up-to-date source for that information. Without further ado, below is the current Zulip roadmap.
4.2 Major projects
There are 2 huge projects that Zulip is working on right now that are too big to have a coherent GitHub issue:
17 Zulip Documentation, Release 1.4.0
• We are working with a world-class designer on a major visual redesign of the Zulip webapp. This will dramati- cally improve the usability of the streams and settings UIs, and make the entire webapp feel like a modern web experience. We plan to make the first release containing this redesign Zulip 2.0, likely in early 2017. • We are writing a new React Native iOS app for Zulip to replace the old iOS app. The new app is progressing rapidly, but is not yet feature complete. We expect it to be in the app store by the end of 2016.
4.3 Core User Experience
• Provide shorter UI/Keyboard sequence to edit the last message • Better drafts management • Make clicking on desktop notifications renarrow properly • Add pretty bubbles for recipients in the compose box • Make right sidebar buddy list UI scale well to large teams • Display stream descriptions more prominently • Add support for managing uploaded files
4.4 Social features
• Add support for showing “user is typing” notifications, at least for private messages • Support lightweight emoji “reactions” • Open graph previews of generic websites • Add a “join Zulip chat” badge for projects that use Zulip to document that nicely
4.5 Real-time sync
The overall goal is to eliminate the few known issues where Zulip does not provide a seamless real-time sync experi- ence. • Notification bot advertisements for new streams don’t handle stream renames • Avatar/name changes don’t propagate to already-sent messages • Advance the pointer / where we load the user to based on unread counts in home view • Fix the known bug where messages could be incorrectly marked as read
4.6 Onboarding issues
This category focuses on issues users experience when installing a new Zulip server, setting up a new Zulip realm, or starting to use Zulip. • Move Zulip’s prompt for permission to display notifications to be manually triggered • Add a mechanism for deleting early test messages (e.g., administrators can hard-delete messages) • Allow customizing emails when inviting new users
18 Chapter 4. Zulip Roadmap Zulip Documentation, Release 1.4.0
4.7 Production installation issues
• Document or better script solution to rabbitmq startup issues • Merge a supported way to use Zulip in Docker in production implementation.
4.8 Administration and management
• Make list of allowed domains web-configurable • Statistics display for realm and server administrators • Keep track of which users added which realm emoji • Add setting to enable any user to add new realm emoji • Make realm filters web-configurable • Improve administrative controls for managing streams • Enhance the LDAP integration and make it web-configurable • Add a SAML integration for Zulip
4.9 Scalability and performance
Scalability and performance are not currently major problems for Zulip; it already scales well to thousands of users and is significantly faster than proprietary alternatives. So, this is not a major focus area for the project. • Make the Zulip Tornado service support horizontal scaling • Make presence system scale well to 10000 users in a realm. • Support running queue workers multithreaded in production to decrease minimum memory footprint • Improve @-mentioning syntax based on stronger unique identifiers
4.10 Technology improvements
• Add support for Zulip running purely on Python 3 • Automatic thumbnailing of uploaded images’ previews to save bandwidth • Upgrade Zulip to use Django 1.10. The patches needed to run Zulip were merged into mainline Django in Django 1.10, so this will mean we don’t need to use a fork of Django anymore. • Upgrade and remove from codebase all unnecessarily vendored JS libraries • Add support for changing users’ email addresses • Migrate from jslint to eslint • Replace the slow closure-compiler based static asset toolchain • Use a modern JavaScript bundler like webpack • Add support for building frontend features in something like React
4.7. Production installation issues 19 Zulip Documentation, Release 1.4.0
4.11 Technical Debt
While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way. • Migrate most web routes to REST API • Split Tornado subsystem into a separate Django app • Refactor zulip.css to be broken into components
4.12 Security
• Add support for 2-factor authentication on all platforms • Add support for stronger security controls for uploaded files (The LOCAL_UPLOADS_DIR file uploads back- end only supports world-readable uploads) • Fix requirement to set a password when creating account via Google • Add a retention policy feature that automatically deletes old messages • Add UI for viewing and cancelling open Zulip invitations
4.13 Testing
• Extend Zulip’s automated test coverage to include all API endpoints • Build automated tests for the client API bindings • Add automated tests for the production upgrade process
4.14 Documentation
• Add an in-app mechanism for updating users about new Zulip features • Significantly expand documentation of the Zulip API and integrating with Zulip. • Write a visual design / frontend style guide for Zulip • Update all screenshots to show the current Zulip UI
4.14.1 Nice to have
• Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developers or system administrators.
4.15 Integrations and bots
Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhook integrations for getting notifications into Zulip, it’d be great to streamline that process and make bots that receive messages just as easy to build.
20 Chapter 4. Zulip Roadmap Zulip Documentation, Release 1.4.0
• Add an outgoing webhook integration system • Make setting up a new integration a smooth flow • Default new incoming webhooks to permissions-limited incoming webhook bots • Change how Zulip displays bot names to distinguish them from human users
4.16 Android app
• Add support for narrowing to @-mentions • Support having multiple Zulip realms open simultaneously
4.17 iOS app
For the new React Native iOS app, the major goal for it is to be released into the app store. Since it is moving quickly, we’re tracking its roadmap via GitHub milestones.
4.18 Server/webapp support for mobile
To support a great mobile experiences, we need to make some improvements in the Zulip server. • Push notifications bouncer service for GCM and APNS • A slick process for doing mobile login without typing your password on your phone • @here mention support (that doesn’t spam people not currently online, i.e. no email/push notifications) • Fix sending messages from mobile web
4.19 Desktop apps
The new cross-platform desktop app is implemented in Electron, and primarily needs work on installer tooling to finish replacing the old app. • Finish releasing the Electron app to replace the old desktop app • Support having multiple Zulip realms open simultaneously
4.20 Community
These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project. • Expand the number of core developers able to do code reviews • Have a successful season with Zulip’s Outreachy participants • Have a successful season with Google Code In.
4.16. Android app 21 Zulip Documentation, Release 1.4.0
22 Chapter 4. Zulip Roadmap CHAPTER 5
Version History
All notable changes to the Zulip server are documented in this file.
5.1 Unreleased
5.2 1.5.0 – 2017-02-06
Highlights: • Completely redesigned the Manage streams interface. • Added support for emoji reactions to messages. • Added a lightbox for viewing images and videos. • Added an extensive user documentation site at /help/. • Added admin setting to auto-linkify certain strings (useful for issue numbers and Git commit IDs). • Upgraded how the main application runs from FastCGI on Django 1.8 to uwsgi and Django 1.10. • Added preliminary support for open graph previews of links (the setting, INLINE_URL_EMBED_PREVIEW, is disabled by default in this release). Full feature Changelog: • Added an emoji picker/browser to the compose box. • Added markdown preview support to the compose box. • Added a new analytics system to track interesting usage statistics. • Added a /stats page with graphs of the analytics data. • Added display of subscriber counts in Manage streams. • Added support for filtering streams in Manage streams.
23 Zulip Documentation, Release 1.4.0
• Added support for setting a stream description on creation. • Added support for copying subscribers from existing streams on creation. • Added several new search/filtering UI elements. • Added UI for deactivating your own Zulip account. • Added support for viewing the raw markdown content of a message. • Added support for deploying Zulip with subdomains for each realm. This entailed numerous changes to ensure a consistent experience. • Added support for (optionally) using PGRoonga to support full-text search in all languages (not just English). • Added AppFollow, GitLab, Google Calendar, GoSquared, HelloSign, Heroku, Librato, MailChimp, Mention, Papertrail, Sentry, Solano Labs, Stripe and Zapier integrations. • Added a webhook integration for GitHub, replacing the deprecated github-services hook. • Normalized the message formatting for all the Zulip Git integrations. • Added support for VMWare Fusion Vagrant provider for faster OSX development. • Added a shields.io style badge for joining a Zulip server. • Added admin setting for which email domains can join a realm. • Added admin setting for controlling who can create streams. • Added admin setting to limit stream creation to older users. • Added a notification when you muted a topic. • Added a new hotkey for muting/unmuting topics. • Added support for testing websockets to the Nagios plugins. • Added a configuration option to disable websockets. • Added support for removing one’s own Zulip account. • Added support for realm admins which auth backends are supported. • Added new organization type concept. This will be used to control whether Zulip is optimized around protecting user privacy vs. administrative control. • Added #streamName syntax for linking to a stream. • Added support for viewing markdown source of messages. • Added setting to always send push notifications. • Added setting to hide private message content in desktop notifications. • Added buttons to download .zuliprc files. • Added italics and strikethrough support in markdown implementation. • Added errors for common installations mistakes (e.g. too little RAM). • Added a new /authors page showing the contributors to the current Zulip version. • Upgraded all Python dependencies to modern versions, including Django 1.10 (all of Zulip’s patches have been merged into mainline). • Increased backend test coverage of Python codebase to 90%. • Increased mypy static type coverage of Python code to 100%.
24 Chapter 5. Version History Zulip Documentation, Release 1.4.0
• Added several new linters (eslint, pep8) and cleaned the codebase. • Optimized the speed of the Zulip upgrade process, especially with Git. • Have peer_add events send user_id, not email. • Fixed problems with rabbitmq when installing Zulip. • Fixed JavaScript not being gzip-compressed properly. • Fixed a major performance bug in the Tornado service. • Fixed a frontend performance bug creating streams in very large realms. • Fixed numerous bugs where strings were not properly tagged for translation. • Fixed several real-time sync bugs, and removed several AJAX calls. Zulip should be more performant than ever before. • Fixed Zulip Tornado service not working with http_proxy set in environment. • Fixed text overflow in stream subscriptions. • Fixed CSS issues with message topic editing. • Fixed several transactionality bugs (e.g. in Huddle creation). • Fixed missed-message email configuration error handling. • Fixed annoying @-mentions in Jira integration. • Fixed various mismatches between frontend and backend markdown implementations. • Fixed various popover-related UI bugs. • Fixed duplicate notifications with multiple open Zulip tabs. • Fixed support for emailing the server administrator about backend exceptions. • Cleaned up the “edit message” form. • Eliminated most of the legacy API endpoints. • Improved typeahead and autocomplete across the application. Highlights include much better handling of many users with similar names. • Improved the color scheme for code blocks. • Improved the message editing UI in several ways. • Improved how dates are displayed in the UI. • Improved default settings for zxcvbn password strength checker. • Upgraded jQuery to the latest 1.12 release. • Made numerous improvements to the development tooling. • Made extensive improvements to code organization. • Restyled all the registration pages to look nicer and be responsive. • Extensively refactored views to use common functions for fetching stream and message objects. • Suppressed @-all mentions being treated as mentions on muted streams. • Documented preliminary design for interactive bot system.
5.2. 1.5.0 – 2017-02-06 25 Zulip Documentation, Release 1.4.0
5.3 1.4.3 - 2017-01-29
• CVE-2017-0881: Users could subscribe to invite-only streams.
5.4 1.4.2 - 2016-09-27
• Upgraded Django to version 1.8.15 (with the Zulip patches applied), fixing a CSRF vulnerability in Django (see https://www.djangoproject.com/weblog/2016/sep/26/security-releases/), and a number of other Django bugs from past Django stable releases that largely affects parts of Django that are not used by Zulip. • Fixed buggy logrotate configuration.
5.5 1.4.1 - 2016-09-03
• Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0. • Fixed local file uploads integration being broken for new 1.4.0 installations.
5.6 1.4.0 - 2016-08-25
• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a major change to how Zulip is installed that we expect will simplify upgrades in the future. • Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached network transfer for loading Zulip. • Added support for using Ubuntu Xenial in production. • Added a powerful and complete realm import/export tool. • Added nice UI for selecting a default language to display settings. • Added UI for searching streams in left sidebar with hotkeys. • Added Semaphore, Bitbucket, and HelloWorld (example) integrations. • Added new webhook-based integration for Trello. • Added management command for creating realms through web UI. • Added management command to send password reset emails. • Added endpoint for mobile apps to query available auth backends. • Added LetsEncrypt documentation for getting SSL certificates. • Added nice rendering of unicode emoji. • Added support for pinning streams to the top of the left sidebar. • Added search box for filtering user list when creating a new stream. • Added realm setting to disable message editing. • Added realm setting to time-limit message editing. Default is 10m. • Added realm setting for default language.
26 Chapter 5. Version History Zulip Documentation, Release 1.4.0
• Added year to timestamps in message interstitials for old messages. • Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authenti- cation methods). • Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulip server. • Added numerous hooks to puppet modules to enable more configurations. • Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.). • Added automatic configuration of postgres/memcached settings based on the server’s available RAM. • Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo. • Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4. • Added support for Name
5.6. 1.4.0 - 2016-08-25 27 Zulip Documentation, Release 1.4.0
• Fixed fastcgi_params to protect against the httpoxy attack. • Fixed bug preventing users with mit.edu emails from registering accounts. • Fixed incorrect settings docs for the email mirror. • Fixed APNS push notification support (had been broken by Apple changing the APNS API). • Fixed some logic bugs in how attachments are tracked. • Fixed unnecessarily resource-intensive rabbitmq cron checks. • Fixed old deployment directories leaking indefinitely. • Fixed need to manually add localhost in ALLOWED_HOSTS. • Fixed display positioning for the color picker on subscriptions page. • Fixed escaping of Zulip extensions to markdown. • Fixed requiring a reload to see newly uploaded avatars. • Fixed @all warning firing even for @all. • Restyled password reset form to look nice. • Improved formatting in reset password links. • Improved alert words UI to match style of other settings. • Improved error experience when sending to nonexistent users. • Portions of integrations documentation are now automatically generated. • Restructured the URLs files to be more readable. • Upgraded almost all Python dependencies to current versions. • Substantially expanded and reorganized developer documentation. • Reorganized production documentation and moved to ReadTheDocs. • Reorganized .gitignore type files to be written under var/ • Refactored substantial portions of templates to support subdomains. • Renamed local_settings.py symlink to prod_settings.py for clarity. • Renamed email-mirror management command to email_mirror. • Changed HTTP verb for create_user_backend to PUT. • Eliminated all remaining settings hardcoded for zulip.com. • Eliminated essentially all remaining hardcoding of mit.edu. • Optimized the performance of all the test suites. • Optimized Django memcached configuration. • Removed old prototype data export tool. • Disabled insecure RC4 cipher in nginx configuration. • Enabled shared SSL session cache in nginx configuration. • Updated header for Zulip static assets to reflect Zulip being open source.
28 Chapter 5. Version History Zulip Documentation, Release 1.4.0
5.7 1.3.13 - 2016-06-21
• Added nearly complete internationalization of the Zulip UI. • Added warning when using @all/@everyone. • Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to. • Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io. • Added menu option to mark all messages in a stream or topic as read. • Added new Attachment model to keep track of uploaded files. • Added caching of virtualenvs in development. • Added mypy static type annotations to about 85% of the Zulip Python codebase. • Added automated test of backend templates to test for regressions. • Added lots of detailed documentation on the Zulip development environment. • Added setting allowing only administrators to create new streams. • Added button to exit the Zulip tutorial early. • Added web UI for configuring default streams. • Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms on a Zulip server. • Fixed email_gateway_password secret not working properly. • Fixed missing helper scripts for RabbitMQ Nagios plugins. • Fixed skipping forward to latest messages (“More messages below” button). • Fixed netcat issue causing Zulip installation to hang on Scaleway machines. • Fixed rendering of /me status messages after message editing. • Fixed case sensitivity of right sidebar fading when compose is open. • Fixed error messages when composing to invalid PM recipients. • Fixed LDAP auth backend not working with Zulip mobile apps. • Fixed erroneous WWW-Authenticate headers with expired sessions. • Changed “coworkers” to “users” in the Zulip UI. • Changed add_default_stream REST API to correctly use PUT rather than PATCH. • Updated the Zulip emoji set (the Android Emoji) to a modern version. • Made numerous small improvements to the Zulip development experience. • Migrated backend templates to the faster Jinja2 templating system. • Migrated development environment setup scripts to tools/setup/. • Expanded test coverage for several areas of the product. • Simplified the API for writing new webhook integrations. • Removed most of the remaining JavaScript global variables.
5.7. 1.3.13 - 2016-06-21 29 Zulip Documentation, Release 1.4.0
5.8 1.3.12 - 2016-05-10
• CVE-2016-4426: Bot API keys were accessible to other users in the same realm. • CVE-2016-4427: Deactivated users could access messages if SSO was enabled. • Fixed a RabbitMQ configuration bug that resulted in reordered messages. • Added expansive test suite for authentication backends and decorators. • Added an option to logout_all_users to delete only sessions for deactivated users.
5.9 1.3.11 - 2016-05-02
• Moved email digest support into the default Zulip production configuration. • Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py. • Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly (e.g. Google Hangouts). • Added new management command to test sending email from Zulip. • Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations. • Added Nagios plugins to the main distribution. • Added ability for realm administrators to manage custom emoji. • Added guide to writing new integrations. • Enabled camo image proxy to fix mixed-content warnings for http images. • Refactored the Zulip puppet modules to be more modular. • Refactored the Tornado event system, fixing old memory leaks. • Removed many old-style /json API endpoints • Implemented running queue processors multithreaded in development, decreasing RAM requirements for a Zulip development environment from ~1GB to ~300MB. • Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant perfor- mance issue in larger realms. • Fixed the disabling of desktop notifications from 1.3.7 for new users. • Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for the realm. • Fixed bugs changing certain settings in administration pages. • Fixed collapsing messages in narrowed views. • Fixed 500 errors when uploading a non-image file as an avatar. • Fixed Jira integration incorrectly not @-mentioning assignee.
30 Chapter 5. Version History Zulip Documentation, Release 1.4.0
5.10 1.3.10 - 2016-01-21
• Added new integration for Travis CI. • Added settings option to control maximum file upload size. • Added support for running Zulip development environment in Docker. • Added easy configuration support for a remote postgres database. • Added extensive documentation on scalability, backups, and security. • Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature. • Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf. • Improved the styling for the Administration page and added tabs. • Substantially improved loading performance on slow networks by enabling GZIP compression on more assets. • Changed the page title in narrowed views to include the current narrow. • Fixed several backend performance issues affecting very large realms. • Fixed bugs where draft compose content might be lost when reloading site. • Fixed support for disabling the “zulip” notifications stream. • Fixed missing step in postfix_localmail installation instructions. • Fixed several bugs/inconveniences in the production upgrade process. • Fixed realm restrictions for servers with a unique, open realm. • Substantially cleaned up console logging from run-dev.py.
5.11 1.3.9 - 2015-11-16
• Fixed buggy #! lines in upgrade scripts.
5.12 1.3.8 - 2015-11-15
• Added options to the Python api for working with untrusted server certificates. • Added a lot of documentation on the development environment and testing. • Added partial support for translating the Zulip UI. • Migrated installing Node dependencies to use npm. • Fixed LDAP integration breaking autocomplete of @-mentions. • Fixed admin panel reactivation/deactivation of bots. • Fixed inaccurate documentation for downloading the desktop apps. • Fixed various minor bugs in production installation process. • Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) who were invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230). • Major preliminary progress towards supporting Python 3.
5.10. 1.3.10 - 2016-01-21 31 Zulip Documentation, Release 1.4.0
5.13 1.3.7 - 2015-10-19
• Turn off desktop and audible notifications for streams by default. • Added support for the LDAP authentication integration creating new users. • Added new endpoint to support Google auth on mobile. • Fixed desktop notifications in modern Firefox. • Fixed several installation issues for both production and development environments. • Improved documentation for outgoing SMTP and the email mirror integration.
32 Chapter 5. Version History CHAPTER 6
Requirements
Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a development environment following these instructions, since then you don’t need to worry about setting up SSL certificates and an authentication mechanism. Or, you can check out the developers’ chatroom (a public, running Zulip instance).
6.1 Server
6.1.1 Hardware Specifications
• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM. For installations with fewer users, 1 CPU and 2GB RAM is sufficient. We strong recommend against installing with less than 2GB of RAM, as you will likely experience out of memory issues installing dependencies. We recommend against using highly CPU-limited servers like the AWS t2 style instances for organizations with a hundreds of users (active or no). • Disk space: You’ll need at least 10GB of free disk space. If you intend to store uploaded files locally rather than on S3 you will likely need more.
6.1.2 Network and Security Specifications
• Incoming HTTPS access (usually port 443, though this is configurable) from the networks where your users are (usually, the public Internet). If you also open port 80, Zulip will redirect users to HTTPS rather than not working when users type e.g. http://zulip.example.com in their browser. If you are using Zulip’s incoming email integration you may also need incoming port 25 open. • Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage inline image previews. If you want to be able to send email from Zulip, you’ll also need outgoing SMTP access to your mail server (using port 587).
33 Zulip Documentation, Release 1.4.0
6.1.3 Operating System
Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running Zulip in production. 64-bit is recommended.
6.1.4 Domain name
You should already have a domain name available for your Zulip production instance. In order to generate valid SSL certificates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update the domains A record to point to your production server.
6.2 Credentials needed
6.2.1 SSL Certificate
• SSL Certificate for the host you’re putting this on (e.g. zulip.example.com). The installation instructions contain documentation for how to get an SSL certificate for free using LetsEncrypt.
6.2.2 Outgoing email
• Email credentials Zulip can use to send outgoing emails to users (e.g. email address confirmation emails during the signup process, missed message notifications, password reminders if you’re not using SSO, etc.). Once you have met these requirements, see full instructions for installing Zulip in production.
34 Chapter 6. Requirements CHAPTER 7
Production Installation
Make sure you want to install a Zulip production server; if you’d instead like to test or develop a new feature, we recommend the Zulip server development environment instead. You will need an Ubuntu system that satisfies the installation requirements. In short, you should have an Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server instance, with at least 4GB RAM, 2 CPUs, and 10 GB disk space. You should also have a domain name available and have updated its DNS record to point to your server.
7.1 Step 0: Subscribe
Please subscribe to low-traffic the Zulip announcements Google Group to get announcements about new releases, security issues, etc.
7.2 Step 1: Install SSL Certificates
Zulip runs over https only and requires ssl certificates in order to work. It looks for the certificates in /etc/ ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note that Zulip uses nginx as its webserver and thus expects a chained certificate bundle If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, just install (or symlink) them into place at the above paths, and move on to the next step.
7.3 Step 2: Download and install latest release
If you haven’t already, download and unpack the latest built server tarball with the following commands: sudo-i # If not already root cd/root wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
35 Zulip Documentation, Release 1.4.0
rm-rf/root/zulip&& mkdir/root/zulip tar-xf zulip-server-latest.tar.gz--directory=/root/zulip--strip-components=1
Then, run the Zulip install script:
/root/zulip/scripts/setup/install
This may take a while to run, since it will install a large number of dependencies. The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue that caused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; please include a copy of that file in any bug reports.
7.4 Step 3: Configure Zulip
Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the manda- tory settings, which are all found under the heading ### MANDATORY SETTINGS. These settings include: • EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation. This will be the domain for which you have DNS A records pointing to this server and for which you configured SSL certificates. • ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who will get support emails. • AUTHENTICATION_BACKENDS: a list of enabled authentication mechanisms. You’ll need to enable at least one authentication mechanism by uncommenting its corresponding line, and then also do any additional config- uration required for that backend as documented in the settings.py file. See the section on Authentication for more detail on the available authentication backends and how to configure them. • EMAIL_*, DEFAULT_FROM_EMAIL, and NOREPLY_EMAIL_ADDRESS: Regardless of which authentication backends you enable, you must provide settings for an outgoing SMTP server so Zulip can send emails when needed (and don’t forget to set email_password in the zulip-secrets.conf file). We highly rec- ommend testing your configuration using su zulip and then /home/zulip/deployments/current/ manage.py send_test_email to confirm your outgoing email configuration is working correctly.
7.5 Step 4: Initialize Zulip database
At this point, you are done doing things as root. To initialize the Zulip database for your production install, run: su zulip-c/home/zulip/deployments/current/scripts/setup/initialize-database
The initialize-database script will report an error if you did not fill in all the mandatory settings from /etc/ zulip/settings.py. It is safe to rerun it after correcting the problem if that happens. This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create an organization in your Zulip installation.
7.6 Step 5: Create a Zulip organization and login
• If you haven’t already, verify that your server can send email using
36 Chapter 7. Production Installation Zulip Documentation, Release 1.4.0
su zulip /home/zulip/deployments/current/manage.py send_test_email [email protected]
You’ll need working outgoing email to complete the setup process. • Run the organization (realm) creation management command :
su zulip # If you weren't already the zulip user /home/zulip/deployments/current/manage.py generate_realm_creation_link
This will print out a secure 1-time use link that allows creation of a new Zulip organization on your server. For most servers, you will only ever do this once, but you can run manage.py generate_realm_creation_link again if you want to host another organization on your Zulip server. • Open the link generated with your web browser. You’ll see the create organization page (screenshot here). Enter your email address and click Create organization. • Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up your organization and initial administrator user (screenshot here). Complete this form and log in! Congratulations! You are logged in as an organization administrator for your new Zulip organization. After getting oriented, we recommend visiting the special “Administration” tab linked to from the upper-right gear menu in the Zulip app to configure important policy settings like how users can join your new organization. By default, your organization will be configured as follows depending on what type of organization you selected: Community Organization: • restricted_to_domain=False: No restriction on user email addresses. • invite_required=True: A user must be invited to join. Corporate Organization: • restricted_to_domain=True: New users must have an email address in the same domain (e.g. @acme.com) as yours. • invite_required=False: No invitation is required to join. Next, you’ll likely want to do one of the following: • Customize your Zulip organization. • Learn about managing a production Zulip server.
7.7 Troubleshooting
If you get an error after scripts/setup/install completes, check /var/log/zulip/errors.log for a traceback, and consult the troubleshooting section for advice on how to debug. If that doesn’t help, please visit the “installation help” stream in the Zulip developers’ chat for realtime help or email [email protected] with the traceback and we’ll try to help you out!
7.7. Troubleshooting 37 Zulip Documentation, Release 1.4.0
38 Chapter 7. Production Installation CHAPTER 8
Troubleshooting
Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, to learn how to use the Supervisor client to monitor and manage services. If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Components section. This will help you understand the many services Zulip uses. If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/. That directory contains one log file for each service, plus errors.log (has all errors), server.log (has logs from the Django and Tornado servers), and workers.log (has combined logs from the queue workers). The section troubleshooting services on this page includes details about how to fix common issues with Zulip services. If you run into additional problems, please report them so that we can update this page! The Zulip installation scripts logs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks from that log.
8.1 Using supervisorctl
To see what Zulip-related services are configured to use Supervisor, look at /etc/supervisor/conf.d/ zulip.conf and /etc/supervisor/conf.d/zulip-db.conf. Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.
8.1.1 Checking status with supervisorctl status
You can check if the zulip application is running using: supervisorctl status
When everything is running as expected, you will see something like this:
39 Zulip Documentation, Release 1.4.0
process-fts-updates RUNNING pid 2194,
˓→uptime1:13:11 zulip-django RUNNING pid 2192,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-0 RUNNING pid 2209,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-1 RUNNING pid 2210,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-2 RUNNING pid 2211,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-3 RUNNING pid 2212,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-4 RUNNING pid 2208,
˓→uptime1:13:11 zulip-tornado RUNNING pid 2193,
˓→uptime1:13:11 zulip-workers:zulip-deliver-enqueued-emails STARTING zulip-workers:zulip-events-confirmation-emails RUNNING pid 2199,
˓→uptime1:13:11 zulip-workers:zulip-events-digest_emails RUNNING pid 2205,
˓→uptime1:13:11 zulip-workers:zulip-events-email_mirror RUNNING pid 2203,
˓→uptime1:13:11 zulip-workers:zulip-events-error_reports RUNNING pid 2200,
˓→uptime1:13:11 zulip-workers:zulip-events-feedback_messages RUNNING pid 2207,
˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_mobile_notifications RUNNING pid 2204,
˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_reminders RUNNING pid 2206,
˓→uptime1:13:11 zulip-workers:zulip-events-signups RUNNING pid 2198,
˓→uptime1:13:11 zulip-workers:zulip-events-slowqueries RUNNING pid 2202,
˓→uptime1:13:11 zulip-workers:zulip-events-user-activity RUNNING pid 2197,
˓→uptime1:13:11 zulip-workers:zulip-events-user-activity-interval RUNNING pid 2196,
˓→uptime1:13:11 zulip-workers:zulip-events-user-presence RUNNING pid 2195,
˓→uptime1:13:11
8.1.2 Restarting services with supervisorctl restart all
After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want to restart the Zulip application. You can restart Zulip using: supervisorctl restart all
8.1.3 Stopping services with supervisorctl stop all
Similarly, you can stop Zulip using:
40 Chapter 8. Troubleshooting Zulip Documentation, Release 1.4.0
supervisorctl stop all
8.2 Troubleshooting services
The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise support the Zulip application: • postgresql • rabbitmq-server • nginx • redis • memcached If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common configuration problems and how to resolve them: • An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running; to fix this, try:
service rabbitmq-server restart
If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configu- ration; you can often correct this by configuring /etc/hosts properly. • If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To fix this, configure them properly and then run:
service nginx restart
Next: Making your Zulip instance awesome.
8.2. Troubleshooting services 41 Zulip Documentation, Release 1.4.0
42 Chapter 8. Troubleshooting CHAPTER 9
Customize Zulip
Once you’ve got Zulip setup, you’ll likely want to configure it the way you like. There are four big things to focus on: 1. Integrations 2. Streams and Topics 3. Notification settings 4. Mobile and desktop apps Lastly, read about Zulip’s other great features, and then enjoy your Zulip installation!
9.1 Integrations
We recommend setting up integrations for the major tools that your team works with. For example, if you’re a software development team, you may want to start with integrations for your version control, issue tracker, CI system, and monitoring tools. Spend time configuring these integrations to be how you like them – if an integration is spammy, you may want to change it to not send messages that nobody cares about (e.g., for the Trac integration, some teams find they only want notifications when new tickets are opened, commented on, or closed, and not every time someone edits the metadata). If Zulip doesn’t have an integration you want, you can add your own! Most integrations are very easy to write, and even more complex integrations usually take less than a day’s work to build. We very much appreciate contributions of new integrations; see the brief integration writing guide. It can often be valuable to integrate your own internal processes to send notifications into Zulip; e.g. notifications of new customer signups, new error reports, or daily reports on the team’s key metrics; this can often spawn discussions in response to the data.
43 Zulip Documentation, Release 1.4.0
9.2 Streams and Topics
If it feels like a stream has too much traffic about a topic only of interest to some of the subscribers, consider adding or renaming streams until you feel like your team is working productively. Second, most users are not used to topics. It can require a bit of time for everyone to get used to topics and start benefitting from them, but usually once a team is using them well, everyone ends up enthusiastic about how much topics make life easier. Some tips on using topics: • When replying to an existing conversation thread, just click on the message, or navigate to it with the arrow keys and hit “r” or “enter” to reply on the same topic • When you start a new conversation topic, even if it’s related to the previous conversation, type a new topic in the compose box • You can edit topics to fix a thread that’s already been started, which can be helpful when onboarding new batches of users to the platform. Third, setting default streams for new users is a great way to get new users involved in conversations before they’ve accustomed themselves with joining streams on their own. You can do so on the /administration page (get there from the gear menu).
9.3 Notification settings
Zulip gives you a great deal of control over which messages trigger desktop notifications; you can configure these extensively in the /#settings page (get there from the gear menu).
9.4 Mobile and desktop apps
Currently, the Zulip Desktop app only supports talking to servers with a properly signed SSL certificate, so you may find that you get a blank screen when you connect to a Zulip server using a self-signed certificate. The iOS app currently available in the App Store doesn’t support Google auth SSO against non-zulip.com servers; there’s a design for how to fix that which wouldn’t be a ton of work to implement. If you are interested in helping out with the Zulip mobile apps, shoot an email to [email protected] and the maintainers can guide you on how to help. For announcements about improvements to the apps, make sure to join the [email protected] list so that you can receive the announcements when these become available.
9.5 All other features
Hotkeys, emoji, search filters, @-mentions, etc. Zulip has lots of great features; make sure your team knows they exist and how to use them effectively.
9.6 Enjoy your Zulip installation!
If you discover things that you wish had been documented, please contribute documentation suggestions either via a GitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation cover everything anyone might want to know about running Zulip in production.
44 Chapter 9. Customize Zulip Zulip Documentation, Release 1.4.0
Next: Maintaining and upgrading Zulip in production.
9.6. Enjoy your Zulip installation! 45 Zulip Documentation, Release 1.4.0
46 Chapter 9. Customize Zulip CHAPTER 10
Secure, maintain, and upgrade
This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including: • Upgrading • Upgrading from a git repository • Backups • Monitoring • Scalability • Management commands You may also want to read this related content: • Security Model
10.1 Upgrading
We recommend reading this entire section before doing your first upgrade. To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.org/ dist/releases/ You also have the option of creating your own release tarballs from a copy of the zulip.git repository using tools/ build-release-tarball. And, starting with Zulip version 1.4, you can upgrade Zulip to a version in a Git repository directly. Next, run as root:
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any database migrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service,
47 Zulip Documentation, Release 1.4.0
which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgrade in advance, we recommend doing upgrades at off hours. Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will require significant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5. Contributions on testing and documenting this process are welcome!
10.1.1 Preserving local changes to configuration files
Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgrade process will overwrite your configuration when it does the puppet apply. You can test whether this will happen assuming no upstream changes to the configuration using scripts/ zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restore your configuration. If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough to handle your setup.
10.1.2 Troubleshooting with the upgrade log
The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs to include output that shows all errors in any bug reports. After the upgrade, we recommend checking /var/log/zulip/errors.log to confirm that your users are not experiencing errors after the upgrade.
10.1.3 Rolling back to a prior version
The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ contain- ing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/ current and /root/zulip as part of the upgrade process. This means that if the new version isn’t working, you can quickly downgrade to the old version by using /home/ zulip/deployments/
10.1.4 Updating settings
If required, you can update your settings by editing /etc/zulip/settings.py and then run /home/zulip/ deployments/current/scripts/restart-server to restart the server.
10.1.5 Applying Ubuntu system updates
While the Zulip upgrade script runs apt-get upgrade, you are responsible for running this on your system on a regular basis between Zulip upgrades to ensure that it is up to date with the latest security patches.
48 Chapter 10. Secure, maintain, and upgrade Zulip Documentation, Release 1.4.0
10.1.6 API and your Zulip URL
To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g. https:// zulip.example.com/api. Our Python API example scripts support this via the --site=https://zulip. example.com argument. The API bindings support it via putting site=https://zulip.example.com in your .zuliprc. Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc file or the en- vironment), but this is not yet documented for some of the integrations (the included integration documentation on /integrations will properly document how to do this for most integrations). We welcome pull requests for integrations that don’t discuss this! Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktop and mobile apps.
10.1.7 Memory leak mitigation
As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automatically restarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise configuration.
10.2 Upgrading from a git repository
Starting with version 1.4, the Zulip server supports doing deployments from a Git repository. To configure this, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf file’s puppet_classes entry, like this:
puppet_classes= zulip::voyager, zulip::static_asset_compiler
Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. You can configure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; by default it uses the main zulip repository (shown below).
[deployment] git_repo_url= https://github.com/zulip/zulip.git
Once that is done (and assuming the currently installed version of Zulip is new enough that this script exists), you can do deployments by running as root:
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git
and Zulip will automatically fetch the relevant branch from the specified repository, build the static assets, and deploy that version. Currently, the upgrade process is slow, but it doesn’t need to be; there is ongoing work on optimizing it.
10.3 Backups
There are several pieces of data that you might want to back up: • The postgres database. That you can back up like any postgres database; we have some example tool- ing for doing that incrementally into S3 using wal-e in puppet/zulip_internal/manifests/ postgres_common.pp (that’s what we use for zulip.com’s database backups). Note that this module isn’t part of the Zulip server releases since it’s part of the zulip.com configuration (see https://github.com/zulip/zulip/ issues/293 for a ticket about fixing this to make life easier for running backups).
10.2. Upgrading from a git repository 49 Zulip Documentation, Release 1.4.0
• Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if you have instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in that directory and you’ll want to back it up. • Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key, all users will need to login again when you setup a replacement server since you won’t be able to verify their cookies; if you lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar file- names are computed using a hash of avatar_salt and user’s email), etc. • The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and it’s definitely lower-priority. If you are interested in backups because you are moving from one Zulip server to another server and can’t transfer a full postgres dump (which is definitely the simplest approach), our draft conversion and export design document may help. The tool is well designed and was tested carefully with dozens of realms as of mid-2016 but is not integrated into Zulip’s regular testing process, and thus it is worth asking on the Zulip developers mailing list whether it needs any minor updates to do things like export newly added tables.
10.3.1 Restore from backups
To restore from backups, the process is basically the reverse of the above: • Install new server as normal by downloading a Zulip release tarball and then using scripts/setup/ install, you don’t need to run the initialize-database second stage which puts default data into the database. • Unpack to /etc/zulip the settings.py and secrets.conf files from your backups. • Restore your database from the backup using wal-e; if you ran initialize-database anyway above, you’ll want to first scripts/setup/postgres-init-db to drop the initial database first. • If you’re using local file uploads, restore those files to the path specified by settings. LOCAL_UPLOADS_DIR and (if appropriate) any logs. • Start the server using scripts/restart-server. This restoration process can also be used to migrate a Zulip installation from one server to another. We recommend running a disaster recovery after you setup backups to confirm that your backups are working; you may also want to monitor that they are up to date using the Nagios plugin at: puppet/zulip_internal/files/ nagios_plugins/check_postgres_backup. Contributions to more fully automate this process or make this section of the guide much more explicit and detailed are very welcome!
10.3.2 Postgres streaming replication
Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files: • puppet/zulip_internal/manifests/postgres_slave.pp • puppet/zulip_internal/manifests/postgres_master.pp • puppet/zulip_internal/files/postgresql/* Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the main puppet/zulip/ tree) would be very welcome!
50 Chapter 10. Secure, maintain, and upgrade Zulip Documentation, Release 1.4.0
10.4 Monitoring
The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available under puppet/ zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs). The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirec- tories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: • check_send_receive_time (sends a test message through the system between two bot users to check that end-to-end message sending works) • check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or the queue workers being behind) • check_queue_worker_errors (checks for errors reported by the queue workers) • check_worker_memory (monitors for memory leaks in queue workers) • check_email_deliverer_backlog and check_email_deliverer_process (monitors for whether outgoing emails are being sent) Database monitoring: • check_postgres_replication_lag (checks streaming replication is up to date). • check_postgres (checks the health of the postgres database) • check_postgres_backup (checks backups are up to date; see above) • check_fts_update_log (monitors for whether full-text search updates are being processed) Standard server monitoring: • check_website_response.sh (standard HTTP check) • check_debian_packages (checks apt repository is up to date) If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it in production are encouraged!
10.5 Scalability
This section attempts to address the considerations involved with running Zulip with a large team (>1000 users). • We recommend using a remote postgres database for isolation, though it is not required. In the following, we discuss a relatively simple configuration with two types of servers: application servers (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers. • You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with just a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get away with half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users, you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoid high rates of memcached misses. • For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. We have not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’s m3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The database
10.4. Monitoring 51 Zulip Documentation, Release 1.4.0
load per user is pretty optimized as long as memcached is working correctly. This has not been tested, but from extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active users using a single large database server without doing anything complicated like sharding the database. • For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (using Postgres streaming replication; see the section on configuring this). Be sure to check out the section on backups if you’re hoping to run a spare application server; in particular you probably want to use the S3 backend for storing user-uploaded files and avatars and will want to make sure secrets are available on the hot spare. • Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There are two issues: you need to share the memcached/Redis/RabbitMQ instance (these should can be moved to a network service shared by multiple servers with a bit of configuration) and the Tornado event system for pushing to browsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other. One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes on a single server, which is also likely the first part of any project to support exchanging events amongst multiple servers. Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping to improve.
10.6 Securing your Zulip server
Zulip’s security model is discussed in a separate document.
10.7 Management commands
Zulip has a large library of Django management commands. To use them, you will want to be logged in as the zulip user and for the purposes of this documentation, we assume the current working directory is /home/zulip/ deployments/current. Below, we should several useful examples, but there are more than 100 in total. We recommend skimming the usage docs (or if there are none, the code) of a management command before using it, since they are generally less polished and more designed for expert use than the rest of the Zulip system.
10.7.1 manage.py shell
You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g., you can do the following to change an email address:
$ /home/zulip/deployments/current/manage.py shell In [1]: user_profile = get_user_profile_by_email("[email protected]") In [2]: do_change_user_email(user_profile, "[email protected]")
manage.py dbshell
This will start a postgres shell connected to the Zulip database.
10.7.2 Grant administrator access
You can make any user a realm administrator on the command line with the knight management command:
52 Chapter 10. Secure, maintain, and upgrade Zulip Documentation, Release 1.4.0
./manage.py knight [email protected]
Creating API super users with manage.py
If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create API super users. To do this, use ./ manage.py knight with the --permission=api_super_user argument. See bots/irc-mirror.py and bots/jabber_mirror.py for further detail on these.
Exporting users and realms with manage.py export
If you need to do an export of a single user or of an entire realm, we have tools in management/ that essentially export Zulip data to the file system. export_single_user.py exports the message history and realm-public metadata for a single Zulip user (includ- ing that user’s received messages as well as their sent messages). A good overview of the process for exporting a single realm when moving a realm to a new server (without moving a full database dump) is in management/export.py. We recommend you read the comment there for words of wisdom on speed, what is and is not exported, what will break upon a move to a new server, and suggested procedure.
10.7.3 Other useful manage.py commands
There are a large number of useful management commands under zerver/management/commands/; you can also see them listed using ./manage.py with no arguments.
10.7. Management commands 53 Zulip Documentation, Release 1.4.0
54 Chapter 10. Secure, maintain, and upgrade CHAPTER 11
Security Model
This section attempts to document the Zulip security model. Since this is new documentation, it likely does not cover every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip development mailing list (or if you think you’ve found a security bug, please report it to [email protected] so we can do a responsible security announcement).
11.1 Secure your Zulip server like your email server
• It’s reasonable to think about security for a Zulip server like you do security for a team email server – only trusted administrators within an organization should have shell access to the server. In particular, anyone with root access to a Zulip application server or Zulip database server, or with access to the zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data (so they can read messages, modify history, etc.). It would be difficult or impossible to avoid this, because the server needs access to the data to support features expected of a group chat system like the ability to search the entire message history, and thus someone with control over the server has access to that data as well.
11.2 Encryption and Authentication
• Traffic between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. • Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks. • The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar, but Zulip also supports password authentication. See the authentication methods documentation for details on Zulip’s available authentication methods.
55 Zulip Documentation, Release 1.4.0
11.2.1 Passwords
Zulip stores user passwords using the standard PBKDF2 algorithm. Password strength is checked and weak passwords are visually discouraged using the popular zxcvbn library. The minimum password strength allowed is controlled by two settings in /etc/zulip/settings.py; PASSWORD_MIN_LENGTH and PASSWORD_MIN_ZXCVBN_QUALITY. The former is self-explanatory; we will explain the latter. Password strength estimation is a complicated topic that we can’t go into great detail on here; we recommend reading the zxvcbn website for details if you are not familiar with password strength analysis. In Zulip’s configuration, a password has quality X if zxcvbn estimates that it would take e^(X * 22) seconds to crack the password with a specific attack scenario. The scenario Zulip uses is one where an the attacker breaks into the Zulip server and steals the hashed passwords; in that case, with a slow hash, the attacker would be able to make roughly 10,000 attempts per second. E.g. a password with quality 0.5 (the default), it would take an attacker about 16 hours to crack such a password in this sort of offline attack. Another important attack scenario is the online attacks (i.e. an attacker sending tons of login requests guessing different passwords to a Zulip server over the web). Those attacks are much slower (more like 10/second without rate limiting), and you should estimate the time to guess a password as correspondingly longer. As a server administrators, you must balance the security risks associated with attackers guessing weak passwords against the usability challenges associated with requiring strong passwords in your organization.
11.3 Messages and History
• Zulip message content is rendered using a specialized Markdown parser which escapes content to protect against cross-site scripting attacks. • Zulip supports both public streams and private (“invite-only”) streams. Any Zulip user can join any public stream in the realm, and can view the complete message history of any public stream without joining the stream. • A private (“invite-only”) stream is hidden from users who are not subscribed to the stream. Users who are not members of a private stream cannot read messages on the stream, send messages to the stream, or join the stream, even if they are a Zulip realm administrator. Users can join private streams only when they are invited. However, any member of a private stream can invite other users to the stream. When a new user joins a private stream, they can see future messages sent to the stream, but they do not receive access to the stream’s message history. • Zulip supports editing the content and topics of messages that have already been sent. As a general philosophy, our policies provide hard limits on the ways in which message content can be changed or undone. In contrast, our policies around message topics favor usefulness (e.g. for conversational organization) over faithfulness to the original. The message editing policy can be configured on the realm administration page. There are three configurations provided out of the box: (i) users cannot edit messages at all, (ii) users can edit any message they have sent, and (iii) users can edit the content of any message they have sent in the last N minutes, and the topic of any message they have sent. In (ii) and (iii), topic edits can also be propagated to other messages with the same original topic, even if those messages were sent by other users. The default setting is (iii), with N = 10. In addition, and regardless of the configuration above, messages with no topic can always be edited to have a topic, by anyone in the organization, and the topic of any message can also always be edited by a realm administrator. Also note that while edited messages are synced immediately to open browser windows, editing messages is not a safe way to redact secret content (e.g. a password) shared unintentionally. Other users may have seen and saved the content of the original message, or have an integration (e.g. push notifications) forwarding all
56 Chapter 11. Security Model Zulip Documentation, Release 1.4.0
messages they receive to another service. Zulip also stores and sends to clients the content of every historical version of a message.
11.4 Users and Bots
• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators have the ability to deactivate and reactivate other human and bot users, delete streams, add/remove administrator privileges, as well as change configuration for the overall realm (e.g. whether an invitation is required to join the realm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages or the messages sent to private streams to which the administrator is not subscribed. However, a Zulip administrator subscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators have administrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to access messages sent to private streams that have bots subscribed, by using the bot’s credentials. In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g. to support auditing functionality). • Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. • To properly remove a user’s access to a Zulip team, it does not suffice to change their password or deactivate their account in the SSO system, since neither of those prevents authenticating with the user’s API key or those of bots the user has created. Instead, you should deactivate the user’s account in the Zulip administration interface (/#administration); this will automatically also deactivate any bots the user had created. • The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s API key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone is lost, in addition to changing passwords, you should rotate the user’s Zulip API key. • Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do including reading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). In particular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm. A likely future feature for Zulip is limited bots that can only send messages. • Certain Zulip bots can be marked as “API super users”; these special bots have the ability to send messages that appear to have been sent by another user (an important feature for implementing integrations like the Jabber, IRC, and Zephyr mirrors).
11.5 User-uploaded content
• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 inte- gration. The URLs of user-uploaded files are secret; if you are using the “local file upload” integration, anyone with the URL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtle attack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access to the file might be leaked to the other server via the Referrer header (see the “Uploads world readable” issue on GitHub). The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented to users don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevant realm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping the URL secret is still important to avoid other users in the Zulip realm from being able to access the file.
11.4. Users and Bots 57 Zulip Documentation, Release 1.4.0
• Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted into the Zulip message feed by other users over HTTPS. • By default, Zulip will provide image previews inline in the body of messages when a message contains a link to an image. You can control this using the INLINE_IMAGE_PREVIEW setting.
11.6 Final notes and security response
If you find some aspect of Zulip that seems inconsistent with this security model, please report it to zulip- [email protected] so that we can investigate and coordinate an appropriate security release if needed. Zulip security announcements will be sent to [email protected], so you should subscribe if you are running Zulip in production.
58 Chapter 11. Security Model CHAPTER 12
Authentication methods
Zulip supports several different authentications methods: • EmailAuthBackend - Email/password authentication. • ZulipLDAPAuthBackend - LDAP username/password authentication. • GoogleMobileOauth2Backend - Google authentication. • GitHubAuthBackend - GitHub authentication. • ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can set REMOTE_USER in Apache. • DevAuthBackend - Only for development, passwordless login as any user. It’s easy to add more; see the docs on python-social-auth below. The setup documentation for most of these is simple enough that we’ve included it inline in /etc/zulip/ settings.py, right above to the settings used to configure them. The remote user authentication backend is more complex since it requires interfacing with a generic third-party authentication system, and so we’ve documented it in detail below.
12.1 Adding additional methods using python-social-auth
The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library. So if you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), you can do this by writing a class similar to GitHubAuthBackend in zproject/backends.py and adding a few settings. Pull requests to add new backends are welcome; they should be tested using the framework in test_auth_backends. py.
59 Zulip Documentation, Release 1.4.0
12.2 Remote User SSO Authentication
Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documents how to configure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable: (0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend as the only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct set depending on whether your SSO system uses email addresses or just usernames in REMOTE_USER. Make sure that you’ve restarted the Zulip server since making this configuration change. (1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read:
puppet_classes= zulip::voyager, zulip::apache_sso
(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install our SSO integration. (3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example and fill in the configuration required for your SSO service to set REMOTE_USER and place your completed configuration file at /etc/apache2/sites-available/zulip-sso.conf zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication, which is useful for testing quickly. You can set it up by doing the following:
/home/zulip/deployments/current/scripts/restart-server cd/etc/apache2/sites-available/ cp zulip-sso.example zulip-sso.conf htpasswd-c/home/zulip/zpasswd username @example.com # prompts for a password
and then continuing with the steps below. (4) Run a2ensite zulip-sso to enable the Apache integration site. (5) Run service apache2 reload to use your new configuration. If Apache isn’t already running, you may need to run service apache2 start instead. Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.
12.2.1 Troubleshooting Remote User SSO
This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not map- ping settings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can often help while debugging to temporarily change the Apache config in /etc/apache2/sites-available/ zulip-sso to listen on all interfaces rather than just 127.0.0.1 as you debug this. It can also be helpful to change /etc/nginx/zulip-include/app.d/external-sso.conf to proxy_pass to a more explicit URL possibly not over HTTPS when debugging. The following log files can be helpful when debugging this setup: • /var/log/zulip/{errors.log,server.log} (the usual places) • /var/log/nginx/access.log (nginx access logs) • /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to “debug” in the Apache config file to make this more verbose) Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summary should help with understanding what’s going on as you try to debug:
60 Chapter 12. Authentication methods Zulip Documentation, Release 1.4.0
• Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends. ZulipRemoteUserBackend, zproject/settings.py configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes https://zulip.example.com/ (a.k.a. the homepage for the main Zulip Django app running behind nginx) redirect to /accounts/login/sso if you’re not logged in. • nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888 apache via the config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the up- stream localhost:8888 defined in /etc/nginx/zulip-include/upstreams). • The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the htpasswd dialogue; you provide correct login information and the request reaches a second Zulip Django app instance that is running behind Apache with with REMOTE_USER set. That request is served by zerver. views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in (sets a cookie) or registers the new user (depending whether they have an account). • After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app sees the cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally via username/password). Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Sugges- tions for how to improve this SSO setup documentation are very welcome!
12.2. Remote User SSO Authentication 61 Zulip Documentation, Release 1.4.0
62 Chapter 12. Authentication methods CHAPTER 13
Postgres database details
13.1 Remote Postgres database
This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting the REMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (see http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for documentation on how to set this up and deploy the certificates) to make the DATABASES configuration in zproject/settings.py work (or override that configuration). If you want to use a remote Postgresql database, you should configure the information about the connection with the server. You need a user called “zulip” in your database server. You can configure these options in /etc/zulip/settings.py (the below descriptions are from the Postgresql documentation): • REMOTE_POSTGRES_HOST: Name or IP address of the remote host • REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are: – disable: I don’t care about security, and I don’t want to pay the overhead of encryption. – allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it. – prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it. – require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want. – verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust. – verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it’s the one I specify. Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf: postgres_password= xxxx
63 Zulip Documentation, Release 1.4.0
Finally, you can stop your database on the Zulip server via: sudo service postgresql stop sudo update-rc.d postgresql disable
In future versions of this feature, we’d like to implement and document how to the remote postgres database server itself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature; if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.
13.2 Debugging postgres database issues
When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY
˓→procpid; which shows the currently running backends and their activity. This is similar to the pg_top output, with the added advantage of showing the complete query, which can be valuable in debugging. To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECT pg_terminate_backend(pid int) as the ‘postgres’ user. The former cancels the backend’s current query and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM to the processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause the database to kill all current connections, roll back any pending transactions, and enter recovery mode.
13.3 Stopping the Zulip postgres database
To start or stop postgres manually, use the pg_ctlcluster command: pg_ctlcluster 9.1[--force] main {start|stop|restart|reload}
By default, using stop uses “smart” mode, which waits for all clients to disconnect before shutting down the database. This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the “fast” mode for shutting down. “Fast” mode is described by the manpage thusly: With the –force option the “fast” mode is used which rolls back all active transactions, disconnects clients immediately and thus shuts down cleanly. If that does not work, shutdown is attempted again in “immediate” mode, which can leave the cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, the postmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failure conditions. This mode should only be used when the machine is about to be shut down. Many database parameters can be adjusted while the database is running. Just modify /etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.
13.4 Debugging issues starting postgres pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check the logs, but you won’t find any information there. pg_ctlcluster runs the following command underneath when it actually goes to start Postgres:
/usr/lib/postgresql/9.1/bin/pg_ctl start-D/var/lib/postgresql/9.1/main-s-o \ '-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
64 Chapter 13. Postgres database details Zulip Documentation, Release 1.4.0
Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However, you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since it does bypass some of the work that pg_ctlcluster does.
13.5 Postgres Vacuuming alerts
The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the autovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUM operation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of the oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios alert, perform a VACUUM in each indicated database as a database superuser (postgres). See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for more details on postgres vacuuming.
13.5. Postgres Vacuuming alerts 65 Zulip Documentation, Release 1.4.0
66 Chapter 13. Postgres database details CHAPTER 14
Development environment installation
14.1 Requirements
The Zulip development environment can be installed on macOS, Windows, and Linux (Ubuntu recommended). You’ll need at least 2GB of available RAM. Installing the Zulip development environment requires downloading several hundred megabytes of dependencies, so you will need an active, reasonably fast, internet connection throughout the entire installation processes. You can configure a proxy if you need one.
14.2 Recommended setup (Vagrant)
For first-time contributors on macOS, Windows, and Ubuntu, we recommend using the Vagrant development environment. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Vagrant adds a bit of overhead to using the Zulip development server, but provides an isolated environment that is easy to install, update, and uninstall. It has been well-tested and performs well.
14.3 Advanced setup (non-Vagrant)
For more experienced contributors, or for first-time contributors who don’t want to or can’t use Vagrant, Zulip supports a wide range of ways to install the Zulip development environment on macOS and Linux (Ubuntu recommended): • On Ubuntu 16.04 Xenial and 14.04 Trusty, you can easily install without using Vagrant. • On other Linux distributions, you’ll need to follow slightly different instructions to install manually. • On macOS and Linux (Ubuntu recommended), you can install using Docker, though support for this remains experimental.
67 Zulip Documentation, Release 1.4.0
Unfortunately, the only supported method to install on Windows is the Vagrant method.
14.4 Slow internet connections
If you have a very slow network connection, however, you may want to avoid using Vagrant (which involves down- loading an Ubuntu virtual machine or Linux Container) and either install directly on Ubuntu 16.04 Xenial or 14.04 Trust, or use the manual install process instead. These options only support Linux. An alternative option if you have poor network connectivity is to rent a cloud server and install the Zulip development environment for remote use. See next section for details.
14.5 Installing remotely
The Zulip development environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to. If you want to run a non-Ubuntu distribution, follow the generic Linux directions.
14.6 Next steps
Once you’ve installed the Zulip development environment, you’ll want to read these documents to learn how to use it: • Using the Development Environment • Testing And if you’ve setup the Zulip development environment on a remote machine, take a look at our tips for developing remotely.
68 Chapter 14. Development environment installation CHAPTER 15
Vagrant environment setup tutorial
This section guides first-time contributors through installing the Zulip development environment on Windows, macOS, and Ubuntu. The recommended method for installing the Zulip development environment is to use Vagrant with VirtualBox on Windows and macOS, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Contents: • Requirements • Step 1: Install Prerequisites • Step 2: Get Zulip code • Step 3: Start the development environment • Step 4: Developing • Troubleshooting & Common Errors • Specifying a proxy If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors. If that doesn’t help, please visit the provision stream in the Zulip developers’ chat for real-time help, or send a note to the Zulip-devel Google group or file an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine
69 Zulip Documentation, Release 1.4.0
15.1 Requirements
Installing the Zulip development environment requires downloading several hundred megabytes of dependencies. You will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxy to access the internet.) • All: 2GB available RAM, Active broadband internet connection. • macOS: macOS (10.11 El Capitan or 10.12 Sierra recommended), Git, VirtualBox, Vagrant. • Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc. • Windows: Windows 64-bit (Win 10 recommended), hardware virtualization enabled (VT-X or AMD-V), ad- ministrator access, Git for Windows (which installs Git BASH), VirtualBox, Vagrant. Don’t see your system listed above? See Advanced setup for details about installing for other Linux and UNIX platforms.
15.2 Step 1: Install Prerequisites
Jump to: • macOS • Ubuntu • Windows
15.2.1 macOS
1. Install Vagrant (1.8.4-1.8.6, do not use 1.8.7). 2. Install VirtualBox (>= 5.1.8) (For a non-free option, but better performance, you can also use VMWare Fusion with the VMWare Fusion Vagrant plugin.) Now you are ready for Step 2: Get Zulip Code.
15.2.2 Ubuntu
The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same. If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code: sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc lxc-templates cgroup-lite redir&
˓→&\ vagrant plugin install vagrant-lxc&&\ vagrant lxc sudoers
For a step-by-step explanation, read on.
70 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
1. Install Vagrant
For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the official Ubuntu repositories. First uninstall any vagrant package you may have installed from the Ubuntu repository:
christie@ubuntu-desktop:~ $ sudo apt-get purge vagrant
Now download and install the .deb package for Vagrant 1.8.6:
christie@ubuntu-desktop:~ $ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb
christie@ubuntu-desktop:~ $ sudo dpkg -i vagrant*.deb
2. Install remaining dependencies
Now install git and lxc-related packages:
christie@ubuntu-desktop:~ $ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
3. Install the vagrant lxc plugin:
christie@ubuntu-desktop:~ $ vagrant plugin install vagrant-lxc Installing the 'vagrant-lxc' plugin. This can take a few minutes... Installed the plugin 'vagrant-lxc (1.2.1)'!
If you encounter an error when trying to install the vagrant-lxc plugin, see this.
4. Configure sudo to be passwordless
Finally, configure sudo to be passwordless when using Vagrant LXC:
christie@ubuntu-desktop:~ $ vagrant lxc sudoers [sudo] password for christie:
If you encounter an error running vagrant lxc sudoers, see this. Now you are ready for Step 2: Get Zulip Code.
15.2.3 Windows 10
1. Install Git for Windows, which installs Git BASH. 2. Install VirtualBox (version >= 5.1.6). 3. Install Vagrant (version 1.8.4-1.8.6, do not use 1.8.7).
15.2. Step 1: Install Prerequisites 71 Zulip Documentation, Release 1.4.0
(Note: While Git BASH is recommended, you may also use Cygwin. If you do, make sure to install default required packages along with git, curl, openssh, and rsync binaries.) After installing, you must run Git BASH as an administrator. Also, you must have hardware virtualization enabled (VT-X or AMD-V) in your computer’s BIOS.
Enable native symlinks
The Zulip code includes symbolic links (symlinks). By default, native Windows symlinks are not enabled in either Git BASH or Cygwin, so you need to do a bit of configuration. You must do this before you clone the Zulip code. In Git for BASH: Open Git BASH as an administrator and run:
$ git config --global core.symlinks true
Now confirm the setting:
$ git config core.symlinks true
If you see true, you are ready for Step 2: Get Zulip Code. Otherwise, if the above command prints false or nothing at all, then symlinks have not been enabled. In Cygwin: Open a Cygwin window as an administrator and do this:
christie@win10 ~ $ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
Next, close that Cygwin window and open another. If you echo $CYGWIN you should see:
christie@win10 ~ $ echo $CYGWIN winsymlinks:native
Now you are ready for Step 2: Get Zulip Code.
15.3 Step 2: Get Zulip Code
If you haven’t already created an ssh key and added it to your GitHub account, you should do that now by following these instructions. 1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged in to GitHub to do this. 2. Open Terminal (macOS/Ubuntu) or Git BASH (Windows; must run as an Administrator) 3. In Terminal/Git BASH, clone your fork:
git clone [email protected]:YOURUSERNAME/zulip.git
This will create a ‘zulip’ directory and download the Zulip code into it. Don’t forget to replace YOURUSERNAME with your git username. You will see something like:
72 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
christie@win10 ~ $ git clone [email protected]:YOURUSERNAME/zulip.git Cloning into 'zulip'... remote: Counting objects: 73571, done. remote: Compressing objects: 100% (2/2), done. remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569 Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done. Resolving deltas: 100% (51448/51448), done. Checking connectivity... done. Checking out files: 100% (1912/1912), done.`
Now you are ready for Step 3: Start the development environment.
15.4 Step 3: Start the development environment
Change into the zulip directory and tell vagrant to start the Zulip development environment with vagrant up. christie@win10 ~ $ cd zulip christie@win10 ~/zulip $ vagrant up
The first time you run this command it will take some time because vagrant does the following: • downloads the base Ubuntu 14.04 virtual machine image (for macOS and Windows) or container (for Ubuntu) • configures this virtual machine/container for use with Zulip, • creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at ~/ zulip • runs the tools/provision script inside the virtual machine/container, which downloads all required depen- dencies, sets up the python environment for the Zulip development server, and initializes a default test database. We call this process “provisioning”. You will need an active internet connection during the entire process. (See Specifying a proxy if you need a proxy to access the internet.) vagrant up can fail while provisioning if your Internet connection is unreliable. To retry, you can use vagrant provision (vagrant up will just boot the guest without provisioning after the first time). Other common issues are documented in the Troubleshooting & Common Errors section. If that doesn’t help, please visit the provision stream in the Zulip developers’ chat for real-time help. On Windows, you will see The system cannot find the path specified. message several times. This is expected behavior and is not an error. Once vagrant up has completed, connect to the development environment with vagrant ssh: christie@win10 ~/zulip $ vagrant ssh
You should see something like this on Windows and macOS:
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Wed May4 21:45:43 UTC 2016
15.4. Step 3: Start the development environment 73 Zulip Documentation, Release 1.4.0
System load: 0.61 Processes: 88 Usage of/: 3.5% of 39.34GB Users logged in:0 Memory usage:7% IP address for eth0: 10.0.2.15 Swap usage:0%
Graph this data and manage this system at: https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud
0 packages can be updated. 0 updates are security updates.
Or something as brief as this in the case of Ubuntu:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)
* Documentation: https://help.ubuntu.com/
Congrats, you’re now inside the Zulip development environment! You can confirm this by looking at the command prompt, which starts with (zulip-venv). Next, start the Zulip server:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ ./tools/run-dev.py
You will see several lines of output starting with something like:
2016-05-04 22:20:33,895 INFO: process_fts_updates starting Recompiling templates 2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates done Validating Django models.py... System check identified no issues (0 silenced).
Django version 1.8 Tornado server is running at http://localhost:9993/ Quit the server with CTRL-C. 2016-05-04 18:20:40,716 INFO Tornado loaded0 event queues in 0.001s 2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds Performing system checks...
And ending with something similar to: http://localhost:9994/webpack-dev-server/ webpack result is served from http://localhost:9991/webpack/ content is served from /srv/zulip webpack: bundle is now VALID. 2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds 2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds
74 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in your browser on your main machine. You should see something like this: The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your browser, check your terminal and you should see something like:
2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) /
˓→(unauth via ?) [04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0 2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via
˓→?) [04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0 2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /
˓→login/ (unauth via ?)
Now you’re ready for Step 4: Developing.
15.5 Step 4: Developing
15.5.1 Where to edit files
You’ll work by editing files on your host machine, in the directory where you cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++, etc.). When you save changes they will be synced automatically to the Zulip development environment on the virtual ma- chine/container. Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how this works are available below. Don’t forget to read through the code style guidelines for details about how to configure your editor for Zulip. For example, indentation should be set to 4 spaces rather than tabs.
15.5.2 Understanding run-dev.py debugging output
It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. See Logging for further details on the run-dev.py console output.
15.5.3 Committing and pushing changes with git
When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (ma- cOS/Ubuntu) or Git BASH (Windows) in the directory where you cloned Zulip on your main machine. If you’re new to working with Git/GitHub, check out our Git & GitHub Guide.
15.5.4 Maintaining the development environment
If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or running tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’ve
15.5. Step 4: Developing 75 Zulip Documentation, Release 1.4.0
recently merged changes to the development environment provisioning process that you need to apply to your devel- opment environment. To update your environment, you’ll need to re-provision your vagrant machine using vagrant provision (this just runs tools/provision from your Zulip checkout inside the Vagrant guest); this should complete in about a minute. After provisioning, you’ll want to (re)start the Zulip development server. If you run into any trouble, the provision stream on chat.zulip.org is a great place to ask for help.
15.5.5 Rebuilding the development environment
If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made to the provisioning process, or because you think something is broken), you can do so using vagrant destroy and then vagrant up. This will usually be much faster than the original vagrant up since the base image is already cached on your machine (it takes about 5 minutes to run with a fast Internet connection).
15.5.6 Shutting down the development environment for use later
To shut down but preserve the development environment so you can use it again later use vagrant halt or vagrant suspend. You can do this from the same Terminal/Git BASH window that is running run-dev.py by pressing ^C to halt the server and then typing exit. Or you can halt vagrant from another Terminal/Git BASH window. From the window where run-dev.py is running:
2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth
˓→via ?) ^C KeyboardInterrupt (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip$ exit logout Connection to 127.0.0.1 closed. christie@win10 ~/zulip
Now you can suspend the development environment:
christie@win10 ~/zulip $ vagrant suspend ==> default: Saving VM state and suspending execution...
If vagrant suspend doesn’t work, try vagrant halt:
christie@win10 ~/zulip $ vagrant halt ==> default: Attempting graceful shutdown of VM...
Check out the Vagrant documentation to learn more about suspend and halt.
15.5.7 Resuming the development environment
When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machine with vagrant ssh and re-start the Zulip server:
76 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
christie@win10 ~/zulip $ vagrant up $ vagrant ssh
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ ./tools/run-dev.py
15.6 Next Steps
Next, read the following to learn more about developing for Zulip: • Git & GitHub Guide • Using the Development Environment • Testing
15.7 Troubleshooting & Common Errors
Below you’ll find a list of common errors and their solutions. If these solutions aren’t working for you or you encounter an issue not documented below, there are a few ways to get further help: • visit the provision stream in the Zulip developers’ chat for real-time help, • send a note to the Zulip-devel Google group, or • file an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine
15.7.1 The box ‘ubuntu/trusty64’ could not be found
If you see the following error when you run vagrant up:
The box 'ubuntu/trusty64' could not be found or could not be accessed in the remote catalog. If this is a private box on HashiCorp's Atlas, please verify you're logged in via `vagrant login`. Also, please double-check the name. The expanded URL and error message are shown below: URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
15.6. Next Steps 77 Zulip Documentation, Release 1.4.0
Then the version of curl that ships with Vagrant is not working on your machine. You are most likely to encounter this error on Windows/Cygwin and macOS. On macOS this error is most likely to occur with Vagrant version 1.8.7 and is a known issue. The solution is to downgrade Vagrant to version 1.8.6 (available here), or to use your system’s version of curl instead of the one that ships with Vagrant:
sudo ln-nsf/usr/bin/curl/opt/vagrant/embedded/bin/curl
On Windows/Cygwin, the fix is simple: replace it with the version from Cygwin. First, determine the location of Cygwin’s curl with which curl:
christie@win10 ~/zulip $ which curl /usr/bin/curl
Now determine the location of Vagrant with which vagrant:
christie@win10 ~/zulip $ which vagrant /cygdrive/c/HashiCorp/Vagrant/bin/vagrant
The path up until /bin/vagrant is what you need to know. In the example above it’s /cygdrive/c/ HashiCorp/Vagrant. Finally, copy Cygwin’s curl to Vagrant embedded/bin directory:
christie@win10 ~/zulip $ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
Now re-run vagrant up and vagrant should be able to fetch the required box file.
15.7.2 os.symlink error
If you receive the following error while running vagrant up:
==> default: Traceback (most recent call last): ==> default: File"./emoji_dump.py", line 75, in
˓→format(name)) ==> default: OSError ==> default: : ==> default: [Errno 71] Protocol error
Then Vagrant was not able to create a symbolic link. First, if you are using Windows, make sure you have run Git BASH (or Cygwin) as an administrator. By default, only administrators can create symbolic links on Windows. Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symbolic links for VirtualBox shared folder. You can check to see that this is enabled for your virtual machine with vboxmanage command. Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settings for this virtual machine with vboxmanage getextradata YOURVMNAME enumerate:
78 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
christie@win10 ~/zulip $ vboxmanage list vms "zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d} christie@win10 ~/zulip $ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1 Key: supported, Value: false
If you see “command not found” when you try to run VBoxManage, you need to add the VirtualBox directory to your path. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\. If vboxmanage enumerate prints nothing, or shows a value of 0 for VBoxInter- nal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command in Terminal/Git BASH/Cygwin: vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/
˓→srv_zulip1
The virtual machine needs to be shut down when you run this command.
15.7.3 Connection timeout on vagrant up
If you see the following error after running vagrant up: default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying...
A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V). If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system. For further information about troubleshooting vagrant timeout errors see this post.
15.7.4 Vagrant up fails with subprocess.CalledProcessError
The vagrant up command basically does the following: • Downloads an Ubuntu image and starts it using a Vagrant provider. • Uses vagrant ssh to connect to that Ubuntu guest, and then runs tools/provision, which has a lot of subcommands that are executed via Python’s subprocess module. These errors mean that one of those subcommands failed. To debug such errors, you can log in to the Vagrant guest machine by running vagrant ssh, which should present you with a standard shell prompt. You can debug interactively by using e.g. cd zulip && ./tools/ provision, and then running the individual subcommands that failed. Once you’ve resolved the problem, you can rerun tools/provision to proceed; the provisioning system is designed to recover well from failures. The zulip provisioning system is generally highly reliable; the most common cause of issues here is a poor network connection (or one where you need a proxy to access the Internet and haven’t configured the development environment to use it.
15.7. Troubleshooting & Common Errors 79 Zulip Documentation, Release 1.4.0
Once you’ve provisioned successfully, you’ll get output like this:
Zulip development environment setup succeeded! (zulip-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$
If the (zulip-venv) part is missing, this is because your installation failed the first time before the Zulip virtualenv was created. You can fix this by just closing the shell and running vagrant ssh again, or using source /srv/ zulip-venv/bin/activate. Finally, if you encounter any issues that weren’t caused by your Internet connection, please report them! We try hard to keep Zulip development environment provisioning free of bugs. pip install fails during vagrant up on Ubuntu
Likely causes are: 1. Networking issues 2. Insufficient RAM. Check whether you’ve allotted at least two gigabytes of RAM, which is the minimum Zulip requires. If not, go to your VM settings and increase the RAM, then restart the VM. npm install errors
The tools/provision script may encounter an error related to npm install that looks something like:
==> default:+ npm install ==> default: Traceback (most recent call last): ==> default: File"/srv/zulip/tools/provision", line 195, in
Usually this error is not fatal. Try connecting to the development environment and re-trying the command from withing the virtual machine: christie@win10 ~/zulip $ vagrant ssh (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ npm install npm WARN optional Skipping failed optional dependency /chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.
˓→0.12
80 Chapter 15. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0
These are just warnings so it is okay to proceed and start the Zulip server.
15.7.5 vagrant-lxc errors
Permissions errors
When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.
NoMethodError
If you see the following error when you try to install the vagrant-lxc plugin:
/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_
˓→by' for nil:NilClass (NoMethodError) from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_
˓→plugin' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin' from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run' from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run' from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in
˓→execute' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute' from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli' from /usr/bin/vagrant:173:in `
And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the issue, which should be fixed when Vagrant 1.8.2 is released. In the meantime, read this post for how to create and apply the patch. It will look something like this: christie@xenial:~ $ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch patching file bundler.rb
15.7. Troubleshooting & Common Errors 81 Zulip Documentation, Release 1.4.0
15.7.6 Permissions errors when running the test suite in LXC
See “Possible testing issues”.
15.8 Specifying a proxy
If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before running Vagrant up. First, install the Vagrant plugin vagrant-proxyconf: vagrant plugin install vagrant-proxyconf.
Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it for your proxy):
HTTP_PROXY http://proxy_host:port HTTPS_PROXY http://proxy_host:port NO_PROXY localhost,127.0.0.1,.example.com
Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed, you’ll need to run vagrant destroy first to clean up the failed installation. You can also change the port on the host machine that Vagrant uses by adding to your ~/. zulip-vagrant-config file. E.g. if you set:
HOST_PORT 9971
(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development server. If you’d like to be able to connect to your development environment from other machines than the VM host, you can manually set the host IP address in the ‘~/.zulip-vagrant-config’ file as well. For example, if you set:
HOST_IP_ADDR 0.0.0.0
(and restart the Vagrant guest), your host IP would be 0.0.0.0, a special value for the IP address that means any IP address can connect to your development server.
82 Chapter 15. Vagrant environment setup tutorial CHAPTER 16
Zulip development environment setup without Vagrant
Contents: • Installing directly on Ubuntu • Installing manually on Linux • Using Docker (experimental)
16.1 Installing directly on Ubuntu
Start by cloning this repository: git clone https://github.com/zulip/zulip.git If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial, you can do that by just running:
# From a clone of zulip.git ./tools/provision source/srv/zulip-venv/bin/activate ./tools/run-dev.py # starts the development server
Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrant destroy to clean up the development environment). Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment, ignoring the parts about vagrant (since you’re not using it).
16.2 Installing manually on Linux
• Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental)
83 Zulip Documentation, Release 1.4.0
• OpenBSD 5.8 (experimental) • Fedora/CentOS common steps • Steps for all systems If you really want to install everything manually, the below instructions should work. Install the following non-Python dependencies: • libffi-dev — needed for some Python extensions • postgresql 9.1 or later — our database (client, server, headers) • nodejs 0.10 (and npm) • memcached (and headers) • rabbitmq-server • libldap2-dev • python-dev • redis-server — rate limiting • tsearch-extras — better text search • libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs
16.2.1 On Debian or Ubuntu systems:
Using the official Ubuntu repositories, PGroonga PPA and tsearch-extras deb package:
Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \ puppet gettext postgresql
# Next, install PGroonga from its PPA sudo add-apt-repository-ys ppa:groonga/ppa sudo apt-get update # On 14.04 sudo apt-get install postgresql-9.3-pgroonga # On 16.04 sudo apt-get install postgresql-9.5-pgroonga
# Next, install Zulip's tsearch-extras postgresql extension # If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras: cd zulip sudo apt-add-repository-ys ppa:tabbott/zulip sudo apt-get update # On 14.04 sudo apt-get install postgresql-9.3-tsearch-extras # On 16.04 sudo apt-get install postgresql-9.5-tsearch-extras
# Otherwise, you can download a .deb directly
84 Chapter 16. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0
# If on 12.04 or wheezy: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-
˓→extras_0.1.2_amd64.deb sudo dpkg-i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
# If on 14.04: wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-
˓→tsearch-extras_0.1.3_amd64.deb sudo dpkg-i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb
# If on 15.04 or jessie: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-
˓→extras_0.1_amd64.deb sudo dpkg-i postgresql-9.4-tsearch-extras_0.1_amd64.deb
# If on 16.04 or stretch wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-
˓→tsearch-extras_0.2_amd64.deb sudo dpkg-i postgresql-9.5-tsearch-extras_0.2_amd64.deb
Alternatively, you can always build the package from tsearch-extras git. Now continue with the All Systems instructions below.
Using the official Zulip PPA (for 14.04 Trusty or 16.04 Xenial):
Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo add-apt-repository ppa:tabbott/zulip sudo apt-get update sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \ puppet gettext tsearch-extras
Now continue with the All Systems instructions below.
16.2.2 On Fedora 22 (experimental):
These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo dnf install libffi-devel memcached rabbitmq-server \ openldap-devel python-devel redis postgresql-server \ postgresql-devel postgresql libmemcached-devel freetype-devel \ nodejs npm yuicompressor closure-compiler gettext
Now continue with the Common to Fedora/CentOS instructions below.
16.2.3 On CentOS 7 Core (experimental):
These instructions are experimental and may have bugs; patches welcome!
16.2. Installing manually on Linux 85 Zulip Documentation, Release 1.4.0
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Add user zulip to the system (not necessary if you configured zulip # as the administrator user during the install process of CentOS 7). useradd zulip
# Create a password for zulip user passwd zulip
# Allow zulip to sudo visudo # Add this line after line `root ALL=(ALL) ALL` zulip ALL=(ALL) ALL
# Switch to zulip user su zulip
# Enable EPEL 7 repo so we can install rabbitmq-server, redis and # other dependencies sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.
˓→rpm
# Install dependencies sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \ python-devel redis postgresql-server postgresql-devel postgresql \ libmemcached-devel wget python-pip openssl-devel freetype-devel \ libjpeg-turbo-devel zlib-devel nodejs yuicompressor \ closure-compiler gettext
# We need these packages to compile tsearch-extras sudo yum groupinstall"Development Tools"
# clone Zulip's git repo and cd into it cd&& git clone https://github.com/zulip/zulip&& cd zulip/
## NEEDS TESTING: The next few DB setup items may not be required at all. # Initialize the postgres db sudo postgresql-setup initdb
# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf
# Change these lines: host all all 127.0.0.1/32 ident host all all::1/128 ident # to this: host all all 127.0.0.1/32 md5 host all all::1/128 md5
Now continue with the Common to Fedora/CentOS instructions below.
16.2.4 On OpenBSD 5.8 (experimental):
These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git
86 Chapter 16. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0
doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \ memcached node libmemcached py-Pillow py-cryptography py-cffi
# Get tsearch_extras and build it (using a modified version which # aliases int4 on OpenBSD): git clone https://github.com/blablacio/tsearch_extras cd tsearch_extras gmake&& sudo gmake install
# Point environment to custom include locations and use newer GCC # (needed for Node modules): export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl" export CXX=eg++
# Create tsearch_data directory: sudo mkdir/usr/local/share/postgresql/tsearch_data
# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/local/share/postgresql/tsearch_data/english.stop sudo touch/usr/local/share/postgresql/tsearch_data/en_us.dict sudo touch/usr/local/share/postgresql/tsearch_data/en_us.affix
Finally continue with the All Systems instructions below.
16.2.5 Common to Fedora/CentOS instructions
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Build and install postgres tsearch-extras module wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.
˓→tar.gz tar xvzf tsearch-extras_0.1.3.tar.gz cd ts2 make sudo make install
# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/share/pgsql/tsearch_data/english.stop sudo touch/usr/share/pgsql/tsearch_data/en_us.dict sudo touch/usr/share/pgsql/tsearch_data/en_us.affix
# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf
# Add this line before the first uncommented line to enable password # auth: host all all 127.0.0.1/32 md5
# Start the services sudo systemctl start redis memcached rabbitmq-server postgresql
16.2. Installing manually on Linux 87 Zulip Documentation, Release 1.4.0
# Enable automatic service startup after the system startup sudo systemctl enable redis rabbitmq-server memcached postgresql
Finally continue with the All Systems instructions below.
16.2.6 All Systems:
Make sure you have followed the steps specific for your platform: • Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental) • OpenBSD 5.8 (experimental) • Fedora/CentOS For managing Zulip’s python dependencies, we recommend using virtualenvs. You must create two virtualenvs. One for Python 2 and one for Python 3. You must also install appropriate python packages in them. You should either install the virtualenvs in /srv, or put symlinks to them in /srv. If you don’t do that, some scripts might not work correctly. You can run tools/setup/setup_venvs.py to do this. This script will create two virtualenvs - /srv/zulip-venv and /srv/zulip-py3-venv. If you want to do it manually, here are the steps: sudo virtualenv /srv/zulip-venv -p python2 # Create a python2 virtualenv sudo chown -R `whoami`:`whoami` /srv/zulip-venv source /srv/zulip-venv/bin/activate # Activate python2 virtualenv pip install --upgrade pip # upgrade pip itself because older versions have known
˓→issues pip install --no-deps -r requirements/py2_dev.txt # install python packages required
˓→for development sudo virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv sudo chown -R `whoami`:`whoami` /srv/zulip-py3-venv source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv pip install --upgrade pip # upgrade pip itself because older versions have known
˓→issues pip install --no-deps -r requirements/py3_dev.txt # install python packages required
˓→for development
Now run these commands:
./tools/install-mypy ./tools/setup/emoji/build_emoji ./scripts/setup/generate_secrets.py --development if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/
˓→tsearch_data/ else sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/ ˓→9.*/tsearch_data/ fi
88 Chapter 16. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0
./scripts/setup/configure-rabbitmq ./tools/setup/postgres-init-dev-db ./tools/do-destroy-rebuild-database ./tools/setup/postgres-init-test-db ./tools/do-destroy-rebuild-test-database ./manage.py compilemessages sudo ./scripts/lib/install-node npm install
To start the development server:
./tools/run-dev.py
. . . and visit http://localhost:9991/.
Proxy setup for by-hand installation
If you are building the development environment on a network where a proxy is required to access the Internet, you will need to set the proxy in the environment as follows: • On Ubuntu, set the proxy environment variables using: export https_proxy=http://proxy_host:port export http_proxy=http://proxy_host:port
• And set the npm proxy and https-proxy using: npm config set proxy http://proxy_host:port npm config set https-proxy http://proxy_host:port
16.3 Using Docker (experimental)
Start by cloning this repository: git clone https://github.com/zulip/zulip.git The docker instructions for development are experimental, so they may have bugs. If you try them and run into any issues, please report them! You can also use Docker to run a Zulip development environment. First, you need to install Docker in your develop- ment machine following the instructions. Some other interesting links for somebody new in Docker are: • Get Started • Understand the architecture • Docker run reference • Dockerfile reference Then you should create the Docker image based on Ubuntu Linux, first go to the directory with the Zulip source code: docker build-t user/zulipdev.
Commit and tag the provisioned images. The below will install Zulip’s dependencies:
16.3. Using Docker (experimental) 89 Zulip Documentation, Release 1.4.0
docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash # /bin/bash /srv/zulip/tools/provision --docker docker ps -af ancestor=user/zulipdev docker commit -m "Zulip installed"
Now you can run the docker server with: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \ /srv/zulip/tools/start-dockers
You’ll want to read the guide for Zulip development to understand how to use the Zulip development. Note that start-dockers automatically runs tools/run-dev.py inside the container; you can then visit http://localhost:9991 to connect to your new Zulip Docker container. To view the container’s run-dev.py console logs to get important debugging information (and e.g. outgoing emails) printed by the Zulip development environment, you can use: docker logs--follow
To restart the server use: docker ps docker restart
To stop the server use: docker ps docker kill
If you want to connect to the Docker instance to run commands (e.g. build a release tarball), you can use: docker ps docker exec -it
If you want to run all the tests you need to start the servers first, you can do it with: docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash $ tools/test-all-docker
You can modify the source code in your development machine and review the results in your browser. Currently, the Docker workflow is substantially less convenient than the Vagrant workflow and less documented; please contribute to this guide and the Docker tooling if you are using Docker to develop Zulip!
90 Chapter 16. Zulip development environment setup without Vagrant CHAPTER 17
Using the Development Environment
Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, the development server homepage just shows a list of the users that exist on the server and you can login as any of them by just clicking on a user. This setup saves time for the common case where you want to test something other than the login process; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION case of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d like to test. While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulip development server encounters. To manually query the Postgres database, run psql zulip for an interactive console. When you make a change, here’s a guide for what you need to do in order to see your change take effect in Develop- ment: • If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’ll just need to reload the browser window to see changes take effect. The Handlebars frontend HTML templates (static/ templates) are automatically recompiled by the tools/compile-handlebars-templates job, which runs as part of tools/run-dev.py. • If you change Python code used by the main Django/Tornado server processes, these services are run on top of Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado servers whenever you save changes to Python code. You can watch this happen in the run-dev.py console to make sure the backend has reloaded. • The Python queue workers will also automatically restart when you save changes. However, you may need to ctrl-C and then restart run-dev.py manually if a queue worker has crashed. • If you change the database schema, you’ll need to use the standard Django migrations process to create and then run your migrations; see the new feature tutorial for an example. Additionally you should check out the detailed testing docs for how to run the tests properly after doing a migration. (In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will take care of running migrations and then cleanly restaring the server for you).
91 Zulip Documentation, Release 1.4.0
92 Chapter 17. Using the Development Environment CHAPTER 18
Developing on a remote machine
The Zulip developer environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
18.1 Connecting to the remote environment
The best way to connect to your server is using the command line tool ssh. • On macOS and Linux/UNIX, ssh is a part of Terminal. • On Windows, ssh comes with Bash for Git. Open Terminal or Bash for Git, and connect with the following:
$ ssh username@host
If you have poor internet connectivity, we recommend using Mosh as it is more reliable over slow or unreliable networks.
18.2 Setting up the development environment
After you have connected to your remote server, you need to install the development environment. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
93 Zulip Documentation, Release 1.4.0
18.3 Running the development server
Once you have set up the development environment, you can start up the development instance of Zulip with the following command in the directory where you cloned Zulip:
./tools/run-dev.py--interface=''
This will start up the Zulip server on port 9991. You can then navigate to http://
18.4 Making changes to code on your remote development server
To see changes on your remote development server, you need to do one of the following: • Edit locally: Clone Zulip code to your computer and then use your favorite editor to make changes. When you want to see changes on your remote Zulip development instance, sync with Git. • Edit remotely: Edit code directly on your remote Zulip development instance using a Web-based IDE (recom- mended for beginners) or a command line editor.
18.4.1 Editing locally
If you want to edit code locally install your favorite text editor. If you don’t have a favorite, here are some suggestions: • atom • emacs • vim • spacemacs • sublime Next, follow our Git and GitHub Guide to clone and configure your fork of zulip on your local computer. Once you have cloned your code locally, you can get to work.
Syncing changes
The easiest way to see your changes on your remote development server is to push them to GitHub and them fetch and merge them from the remote server. For more detailed instructions about how to do this, see our Git & GitHub Guide. In brief, the steps are as follows. On your local computer: 1. Open Terminal (macOS/Linux) or Git for BASH. 2. Change directory to where you cloned Zulip (e.g. cd zulip). 3. Use git add and git commit to stage and commit your changes (if you haven’t already). 4. Push your commits to GitHub with git push origin branchname.
94 Chapter 18. Developing on a remote machine Zulip Documentation, Release 1.4.0
Be sure to replace branchname with the name of your actual feature branch. Once git push has completed successfully, you are ready to fetch the commits from your remote development instance: 1. In Terminal or Git BASH, connect to your remote development instance with ssh user@host. 2. Change to the zulip directory (e.g., cd zulip). 3. Fetch new commits from GitHub with git fetch origin. 4. Change to the branch you want to work on with git checkout branchname. 5. Merge the new commits into your branch with git merge origin/branchname.
18.4.2 Editing remotely
Web-based IDE
If you are relatively new to working on the command line, or just want to get started working quickly, we recommend web-based IDE Codeanywhere. To setup Codeanywhere for Zulip: 1. Create a Codeanywhere account and log in. 2. Create a new SFTP-SSH project. Use Public key for authentication. 3. Click GET YOUR PUBLIC KEY to get the new public key that Codeanywhere generates when you create a new project. Add this public key to ~/.ssh/authorized_keys on your remote development instance. 4. Once you’ve added the new public key to your remote development instance, click CONNECT.
18.4. Making changes to code on your remote development server 95 Zulip Documentation, Release 1.4.0
Now your workspace should look similar this:
Command line editors
Another way to edit directly on the remote development server is with a command line text editor on the remote machine. Two editors often available by default on Linux systems are: • Nano: A very simple, beginner-friendly editor. However, it lacks a lot of features useful for programming, such as syntax highlighting, so we only recommended it for quick edits to things like configuration files. Launch by running command nano
18.4.3 Next steps
Next, read the following to learn more about developing for Zulip: • Git & GitHub Guide
96 Chapter 18. Developing on a remote machine Zulip Documentation, Release 1.4.0
• Using the Development Environment • Testing
18.4. Making changes to code on your remote development server 97 Zulip Documentation, Release 1.4.0
98 Chapter 18. Developing on a remote machine CHAPTER 19
Writing a new integration
Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to making integrating with Zulip and getting you integration merged upstream so everyone else can benefit from it as easy as possible while maintaining the high quality of the Zulip integrations library. On this page you’ll find: • An overview of the different types of integrations possible with Zulip. • General advice for writing integrations. • Details about writing webhook integrations. • Details about writing Python script and plugin integrations. • A guide to documenting your integration. •A detailed walkthrough of a simple “Hello World” integration. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help writing integration, please email [email protected], open an issue, or submit a pull request to share your ideas!
19.1 Types of integrations
We have several different ways that we integrate with 3rd party products, ordered here by which types we prefer to write: 1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting content to a particular URI on our site with data about the event. For these, you usually just need to create a new python package in the zerver/webhooks/ directory. You can easily find recent commits adding new integrations to crib from via git log zerver/webhooks/. 2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (by shelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrations in our API release tarballs (by writing the integration in api/integrations).
99 Zulip Documentation, Release 1.4.0
3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existing software. These are often more work, but for some products are the only way to integrate with the product at all.
19.2 General advice
• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g. emoji, markdown emphasis, @-mentions, or !avatar(email)). • Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes for much better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for all messages; for an integration like Nagios, put the service in the topic. • Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to provid- ing options for triggering Zulip messages only for certain message types, certain projects, or sending different messages to different streams/topics, to make it easy for teams to configure the integration to support their workflow. • Consistently capitalize the name of the integration in the documentation and the Client name the way the vendor does. It’s OK to use all-lower-case in the implementation. • Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use – sometimes the right API is just not properly documented. • A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via your local Zulip development environment. This enables you to do end-to-end testing with live data from the service you’re integrating and can help you spot why something isn’t working or if the service is using custom HTTP headers.
19.3 Webhook integrations
A webhook allows a third-party service to push data to you when something happens. It’s different from making a REST API call, where you send a request to the service’s API and wait for a response. With a webhook, the third- party service sends you an HTTP POST when it has something for you. Your webhook integration defines the URI the service uses to communicate with Zulip, and handles that incoming data. New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use the right process. Here’s how we recommend doing it: • First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’re integrating. You can use these captured payloads to create a set of test fixtures for your integration under zerver/fixtures. • Then write a draft webhook handler under zerver/webhooks/; there are a lot of examples in that directory. We recommend templating off a short one (like stash or zendesk), since the longer ones usually just have more complex parsing which can obscure what’s common to all webhook integrations. In addition to writing the integration itself, you’ll need to create Integration object and add it to WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py; search for webhook in that file to find the existing ones (and please add yours in the alphabetically correct place). • Then write a test for your fixture in the tests.py file in the zerver/webhooks/mywebhook directory. You can now iterate on debugging the tests and webhooks handler until they work, all without ever needing to post directly from the service you’re integrating with to your Zulip development machine. You can run just the tests for one integration like this:
100 Chapter 19. Writing a new integration Zulip Documentation, Release 1.4.0
test-backend zerver/webhooks/pagerduty/
See this guide for more details on the Zulip test runner. • Once you’ve gotten your webhook working and passing a test, capture payloads for the other common types of posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast. Webhook integration tests should all use fixtures (as opposed to contacting the service), since otherwise the tests can’t run without Internet access and some sort of credentials for the service. • Finally, write documentation for the integration; there’s a detailed guide below. See the Hello World webhook Walkthrough below for a detailed look at how to write a simple webhook.
19.3.1 Files that need to be created
Select a name for your webhook and use it consistently. The examples below are for a webhook named ‘MyWebHook’. • static/images/integrations/logos/mywebhook.png: An image to represent your integration in the user interface. Generally this Should be the logo of the platform/server/product you are integrating. See Documenting your integration for details. • static/images/integrations/mywebbook/001.png: A screen capture of your integration for use in the user interface. You can add as many images as needed to effectively document your webhook integration. See Documenting your integration for details. • zerver/fixtures/mywebhook/mywebhook_messagetype.json: Sample json payload data used by tests. Add one fixture file per type of message supported by your integration. See Testing and writing tests for details. • zerver/webhooks/mywebhook/__init__.py: Empty file that is obligatory part of every python pack- age. Remember to git add it. • zerver/webhooks/mywebhook/view.py: Includes the main webhook integration function including any needed helper functions. • zerver/webhooks/mywebhook/tests.py: Add tests for your webbook. See Testing and writing tests for details. • zerver/webhooks/mywebhook/doc.html: Add end-user documentation. See Documenting your inte- gration for details.
19.3.2 Files that need to be updated
• zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONS to register it. This will automatically register a url for the webhook of the form api/v1/external/mywebhook and associate with the function called api_mywebhook_webhook in zerver/webhooks/mywebhook/ view.py.
19.4 Python script and plugin integrations
For plugin integrations, usually you will need to consult the documentation for the third party software in order to learn how to write the integration. But we have a few notes on how to do these: • You should always send messages by POSTing to URLs of the form https://zulip.example.com/v1/ messages/.
19.4. Python script and plugin integrations 101 Zulip Documentation, Release 1.4.0
• We usually build Python script integration with (at least) 2 files: zulip_foo_config.py containing the configuration for the integration including the bots’ API keys, plus a script that reads from this configuration to actually do the work (that way, it’s possible to update the script without breaking users’ configurations). • Be sure to test your integration carefully and document how to install it (see notes on documentation below). • You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identify the integration and version number, separated by a slash. If possible, you should collect platform information and include that in ()s after the version number. Some examples of ideal UAs are:
ZulipDesktop/0.7.0 (Ubuntu; 14.04) ZulipJenkins/0.1.0 (Windows; 7.2) ZulipMobile/0.5.4 (Android; 4.2; maguro)
19.5 Documenting your integration
Every Zulip integration must be documented in zerver/webhooks/mywebhook/doc.html. Usually, this in- volves a few steps: • Add text explaining all of the steps required to setup the integration, including what URLs to use, etc. If there are any screens in the product involved, take a few screenshots with the input fields filled out with sample values in order to make the instructions really easy to follow. For the screenshots, use a bot with a name like “GitHub Bot”, an email address for the bot like [email protected], and an obviously fake API key like abcdef123456790. • Make sure you’ve added your integration to zerver/lib/integrations.py; this results in your integra- tion appearing on the /integrations page. You’ll need to add a logo image for your integration under the static/images/integrations/logos/
./manage.py send_webhook_fixture_message \ --fixture=zerver/fixtures/pingdom/pingdom_imap_down_to_up.json \ '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'
When generating the screenshot of a sample message, give your test bot a nice name like “GitHub Bot”, use the project’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not just the message body. When writing documentation for your integration, be sure to use the {{ external_api_uri }} template variable, so that your integration documentation will provide the correct URL for whatever server it is deployed on. If special configuration is required to set the SITE variable, you should document that too, inside an {% if api_site_required %} check.
19.6 Hello World webhook Walkthrough
Below explains each part of a simple webhook integration, called Hello World. This webhook sends a “hello” message to the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receives in the http request. Use this walkthrough to learn how to write your first webhook integration.
102 Chapter 19. Writing a new integration Zulip Documentation, Release 1.4.0
19.6.1 Step 0: Create fixtures
The first step in creating a webhook is to examine the data that the service you want to integrate will be sending to Zulip. You can use requestb.in or a similar tool to capture webhook payload(s) from the service you are integrating. Examin- ing this data allows you to do two things: 1. Determine how you will need to structure your webhook code, including what message types your integration should support and how; and, 2. Create fixtures for your webhook tests. A test fixture is a small file containing test data, one for each test. Fixtures enable the testing of webhook integration code without the need to actually contact the service being integrated. Because Hello World is a very simple webhook that does one thing, it requires only one fixture, zerver/ fixtures/helloworld/helloworld_hello.json:
{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe", }
When writing your own webhook integration, you’ll want to write a test function for each distinct message condition your webhook supports. You’ll also need a corresponding fixture for each of these tests. See Step 3: Create tests or Testing for further details.
19.6.2 Step 1: Initialize your webhook python package
In the zerver/webhooks/ directory, create new subdirectory that will contain all of corresponding code. In our example it will be helloworld. The new directory will be a python package, so you have to create an empty __init__.py file in that directory via e.g. touch zerver/webhooks/helloworld/__init__.py.
19.6.3 Step 2: Create main webhook code
The majority of the code for your webhook integration will be in a single python file, zerver/webhooks/ mywebhook/view.py. The Hello World integration is in zerver/webhooks/helloworld/view.py: from __future__ import absolute_import from django.utils.translation import ugettext as _ from zerver.lib.actions import check_send_message from zerver.lib.response import json_success, json_error from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view from zerver.lib.validator import check_dict, check_string from zerver.models import Client, UserProfile from django.http import HttpRequest, HttpResponse from typing import Dict, Any, Iterable, Optional, Text
@api_key_only_webhook_view('HelloWorld') @has_request_variables def api_helloworld_webhook(request, user_profile, client, payload=REQ(argument_type='body'),
19.6. Hello World webhook Walkthrough 103 Zulip Documentation, Release 1.4.0
stream=REQ(default='test'), topic=REQ(default='Hello World')): # type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]],
˓→Text, Optional[Text]) -> HttpResponse
# construct the body of the message body='Hello! I am happy to be here! :smile:'
# try to add the Wikipedia article of the day # return appropriate error if not successful try: body_template=' \nThe Wikipedia featured article for today is **[{featured_ ˓→title}]({featured_url})**' body+= body_template.format( **payload) except KeyError as e: return json_error(_("Missing key {} in JSON").format(str(e)))
# send the message check_send_message(user_profile, client,'stream', [stream], topic, body)
# return json result return json_success()
The above code imports the required functions and defines the main webhook function api_helloworld_webhook, decorating it with api_key_only_webhook_view and has_request_variables. You must pass the name of your webhook to the api_key_only_webhook_view decorator. Here we have used HelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating in camel case, spelled as the product spells its own name (except always first letter upper-case). You should name your webhook function as such api_webhookname_webhook where webhookname is the name of your webhook and is always lower-case. At minimum, the webhook function must accept request (Django HttpRequest object), user_profile (Zulip’s user object), and client (Zulip’s analogue of UserAgent). You may also want to define additional parameters using the REQ object. In the example above, we have defined payload which is populated from the body of the http request, stream with a default of test (available by default in the Zulip development environment), and topic with a default of Hello World. The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate your webhook functions. In the body of the function we define the body of the message as Hello! I am happy to be here! :smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as pro- vided by the json payload. If the json payload does not include data for featured_title and featured_url we catch a KeyError and use json_error to return the appropriate information: a 400 http status code with relevant details. Then we send a public (stream) message with check_send_message which will validate the message and then send it. Finally, we return a 200 http status with a JSON format success message via json_success().
104 Chapter 19. Writing a new integration Zulip Documentation, Release 1.4.0
19.6.4 Step 3: Create an api endpoint for the webhook
In order for a webhook to be externally available, it must be mapped to a url. This is done in zerver/lib/ integrations.py. Look for the lines beginning with:
WEBHOOK_INTEGRATIONS=[
And you’ll find the entry for Hello World:
WebhookIntegration('helloworld', display_name='Hello World'),
This tells the Zulip api to call the api_helloworld_webhook function in zerver/webhooks/ helloworld/view.py when it receives a request at /api/v1/external/helloworld. This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page using static/images/ integrations/logos/helloworld.png as its icon. At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough code to test your integration. First, get an API key from the Your bots section of your Zulip user’s Settings page. If you haven’t created a bot already, you can do that there. Then copy its API key and replace the placeholder “
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/fixtures/helloworld/helloworld_hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key=
After which you should see something similar to:
2016-07-07 15:06:59,187 INFO 127.0.0.1 POST 200 143ms (mem:6ms/13) (md:
˓→43ms/1) (db: 20ms/9q) (+start: 147ms)/api/v1/external/helloworld (helloworld-
˓→[email protected] via ZulipHelloWorldWebhook)
Using curl: curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn
˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://
˓→localhost:9991/api/v1/external/helloworld\?api_key\=
After which you should see:
{"msg":"","result":"success"}
Using either method will create a message in Zulip:
19.6. Hello World webhook Walkthrough 105 Zulip Documentation, Release 1.4.0
19.6.5 Step 4: Create tests
Every webhook integration should have a corresponding test file: zerver/webhooks/mywebhook/tests.py. The Hello World integration’s tests are in zerver/webhooks/helloworld/tests.py You should name the class
# Note: Include a test function per each distinct message condition your
˓→integration supports def test_hello_message(self): # type: () -> None expected_subject= u"Hello World"; expected_message= u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/ ˓→Marilyn_Monroe)**";
# use fixture named helloworld_hello self.send_and_test_stream_message('hello', expected_subject, expected_message, content_type="application/x-www-form-
˓→urlencoded")
def get_body(self, fixture_name): # type: (Text) -> Text return self.fixture_data("helloworld", fixture_name, file_type="json")
In the above example, STREAM_NAME, URL_TEMPLATE, and FIXTURE_DIR_NAME refer to class attributes from the base class, WebhookTestCase. These are needed by send_and_test_stream_message to determine how to execute your test. When writing tests for your webhook, you’ll want to include one test function (and corresponding fixture) per each distinct message condition that your integration supports. If, for example, we added support for sending a goodbye message to our Hello World webhook, we would add another test function to HelloWorldHookTests class called something like test_goodbye_message:
def test_goodbye_message(self): # type: () -> None expected_subject= u"Hello World"; expected_message= u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
# use fixture named helloworld_goodbye self.send_and_test_stream_message('goodbye', expected_subject, expected_
˓→message, content_type="application/x-www-form-
˓→urlencoded")
As well as a new fixture helloworld_goodbye.json in zerver/fixtures/helloworld/:
{ "featured_title":"Goodbye",
106 Chapter 19. Writing a new integration Zulip Documentation, Release 1.4.0
"featured_url":"https://en.wikipedia.org/wiki/Goodbye", }
Once you have written some tests, you can run just these new tests from within the Zulip development environment with this command:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend zerver/webhooks/helloworld
(Note: You must run the tests from the top level of your development directory. The standard location in a Vagrant environment is /srv/zulip. If you are not using Vagrant, use the directory where you have your development environment.) You will see some script output and if all the tests have passed, you will see:
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_goodbye_message Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_hello_message DONE!
19.6.6 Step 5: Create documentation
Next, we add end-user documentation for our webhook integration. You can see the existing examples at https: //zulipchat.com/integrations or by accessing /integrations in your Zulip development environemtn. There are two parts to the end-user documentation on this page. The first is the lozenge in the grid of integrations, showing your integration logo and name, which links to the full documentation. This is generated automatically once you’ve registered the integration in WEB- HOOK_INTEGRATIONS in zerver/lib/integrations.py, and supports some customization via options to the WebhookIntegration class. Second, you need to write the actual documentation content in zerver/webhooks/mywebhook/doc.html.
Learn how Zulip integrations work with this simple Hello World example!
The Hello World webhook will use the test stream, which is created by default in the Zulip development environment. If you are running Zulip in production, you should make sure this stream exists.
Next, on your Zulip settings page, create a Hello World bot. Construct the URL for the Hello World bot using the API key and stream name: {{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&stream=test
˓→
To trigger a notication using this webhook, use `send_webhook_fixture_message`
˓→from the Zulip command line:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/fixtures/helloworld/helloworld_hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key='
Or, use curl:
19.6. Hello World webhook Walkthrough 107 Zulip Documentation, Release 1.4.0
curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":˓→"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }'
˓→http://localhost:9991/api/v1/external/helloworld\?api_key\=
Congratulations! You're done!
Your messages may look like:
These documentation blocks should fall alphabetically. For the integration-lozenge div this happens automat- ically when the html is generated. For the integration-instructions div, we have added the div between the blocks for GitHub and Hubot, respectively. See Documenting your integration for further details, including how to easily create the message screenshot.
19.6.7 Step 5: Preparing a pull request to zulip/zulip
When you have finished your webhook integration and are ready for it to be available in the Zulip product, follow these steps to prepare your pull request: 1. Run tests including linters and ensure you have addressed any issues they report. See Testing for details. 2. Read through Code styles and conventions and take a look through your code to double-check that you’ve followed Zulip’s guidelines. 3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control for tips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll want to squash your changes into a single commit and include a good, clear commit message. 4. Push code to your fork. 5. Submit a pull request to zulip/zulip. If you would like feedback on your integration as you go, feel free to submit pull requests as you go, prefixing them with [WIP].
108 Chapter 19. Writing a new integration CHAPTER 20
Writing a new application feature
The changes needed to add a new feature will vary, of course, but this document provides a general outline of what you may need to do, as well as an example of the specific steps needed to add a new feature: adding a new option to the application that is dynamically synced through the data system in real-time to all browsers the user may have open.
20.1 General Process in brief
20.1.1 Adding a field to the database
Update the model: The server accesses the underlying database in zerver/ models.py. Add a new field in the appropriate class. Create and run the migration: To create and apply a migration, run:
./manage.py makemigrations ./manage.py migrate
Test your changes: Once you’ve run the migration, restart memcached on your development server (/etc/init. d/memcached restart) and then restart run-dev.py to avoid interacting with cached objects.
20.1.2 Backend changes
Database interaction: Add any necessary code for updating and interacting with the database in zerver/lib/ actions.py. It should update the database and send an event announcing the change. Application state: Modify the fetch_initial_state_data and apply_events functions in zerver/ lib/actions.py to update the state based on the event you just created. Backend implementation: Make any other modifications to the backend required for your change. New views: Add any new application views to zerver/urls.py. This includes both views that serve HTML (new pages on Zulip) as well as new API endpoints that serve JSON-formatted data.
109 Zulip Documentation, Release 1.4.0
Testing: At the very least, add a test of your event data flowing through the system in test_events.py.
20.1.3 Frontend changes
JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact files you may need to change depend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it to static/ js/server_events.js. CSS: The primary CSS file is static/styles/zulip.css. If your new feature requires UI changes, you may need to add additional CSS to this file. Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. For JavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part of the build/deploy process. Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackbox tests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. The unit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information on writing and running tests see the testing documentation.
20.2 Example Feature
This example describes the process of adding a new setting to Zulip: a flag that restricts inviting new users to admins only (the default behavior is that any user can invite other users). It is based on an actual Zulip feature, and you can review the original commit in the Zulip git repo. (Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the migration format has changed.)
20.2.1 Update the model
First, update the database and model to store the new setting. Add a new boolean field, invite_by_admins_only, to the Realm model in zerver/models.py.
--- a/zerver/models.py +++ b/zerver/models.py @@ -139,6 +139,7 @@ class Realm(ModelReprMixin, models.Model): restricted_to_domain = models.BooleanField(default=True) # type: bool invite_required = models.BooleanField(default=False) # type: bool + invite_by_admins_only = models.BooleanField(default=False) # type: bool create_stream_by_admins_only = models.BooleanField(default=False) # type: bool mandatory_topics = models.BooleanField(default=False) # type: bool
20.2.2 Create the migration
Create the migration file: ./manage.py makemigrations. Make sure to commit the generated file to git: git add zerver/migrations/NNNN_realm_invite_by_admins_only.py (NNNN is a number that is equal to the number of migrations.) If you run into problems, the Django migration documentation is helpful.
110 Chapter 20. Writing a new application feature Zulip Documentation, Release 1.4.0
20.2.3 Test your migration changes
Apply the migration: ./manage.py migrate Output:
shell $ ./manage.py migrate Operations to perform: Synchronize unmigrated apps: staticfiles, analytics, pipeline Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,
˓→auth, contenttypes Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying zerver.0026_realm_invite_by_admins_only... OK
20.2.4 Handle database interactions
Next, we will move on to implementing the backend part of this feature. Like typical apps, we will need our backend to update the database and send some response to the client that made the request. Beyond that, we need to orchestrate notifications to other clients (or other users, if you will) that our setting has changed. Clients find out about settings through two closely related code paths. When a client first contacts the server, the server sends the client its initial state. Subsequently, clients subscribe to “events,” which can (among other things) indicate that settings have changed. For the backend piece, we will need our action to make a call to send_event to send the event to clients that are active. We will also need to modify fetch_initial_state_data so that future clients see the new changes. Anyway, getting back to implementation details... In zerver/lib/actions.py, create a new function named do_set_realm_invite_by_admins_only. This function will update the database and trigger an event to notify clients when this setting changes. In this case there was an existing realm|update event type which was used for setting similar flags on the Realm model, so it was possible to add a new property to that event rather than creating a new one. The property name matches the database field to make it easy to understand what it indicates. The second argument to send_event is the list of users whose browser sessions should be notified. Depending on the setting, this can be a single user (if the setting is a personal one, like time display format), only members in a particular stream or all active users in a realm. :
# zerver/lib/actions.py
def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only): realm.invite_by_admins_only= invite_by_admins_only realm.save(update_fields=['invite_by_admins_only']) event= dict( type="realm", op="update", property='invite_by_admins_only', value=invite_by_admins_only, ) send_event(event, active_user_ids(realm)) return {}
20.2. Example Feature 111 Zulip Documentation, Release 1.4.0
20.2.5 Update application state
You then need to add code that will handle the event and update the application state. In zerver/lib/actions. py update the fetch_initial_state and apply_events functions. :
def fetch_initial_state_data(user_profile, event_types, queue_id): # ... state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`
In this case you don’t need to change apply_events because there is already code that will correctly handle the realm update event type: :
def apply_events(state, events, user_profile): for event in events: # ... elif event['type'] =='realm': field='realm_'+ event['property'] state[field]= event['value']
20.2.6 Add a new view
You then need to add a view for clients to access that will call the newly-added actions.py code to update the database. This example feature adds a new parameter that should be sent to clients when the application loads and be accessible via JavaScript, and there is already a view that does this for related flags: update_realm. So in this case, we can add out code to the existing view instead of creating a new one. :
# zerver/views/__init__.py
def home(request): # ... page_params= dict( # ... realm_invite_by_admins_only= register_ret['realm_invite_by_admins_only'], # ... )
Since this feature also adds a checkbox to the admin page, and adds a new property the Realm model that can be modified from there, you also need to make changes to the update_realm function in the same file: :
# zerver/views/realm.py
def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None), restricted_to_domain=REQ(validator=check_bool, default=None), invite_required=REQ(validator=check_bool, default=None), ...more arguments):
# ...
if invite_by_admins_only is not None and realm.invite_by_admins_only != invite_by_admins_only: do_set_realm_invite_by_admins_only(realm, invite_by_admins_only) data['invite_by_admins_only']= invite_by_admins_only
Then make the required front end changes: in this case a checkbox needs to be added to the admin page (and its value added to the data sent back to server when a realm is updated) and the change event needs to be handled on the client.
112 Chapter 20. Writing a new application feature Zulip Documentation, Release 1.4.0
To add the checkbox to the admin page, modify the relevant template, static/templates/admin_tab. handlebars (omitted here since it is relatively straightforward). Then add code to handle changes to the new form control in static/js/admin.js.: var url = "/json/realm"; var new_invite_by_admins_only = $("#id_realm_invite_by_admins_only").prop("checked"); data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only); channel.patch({ url: url, data: data, success: function (data) { # ... if (data.invite_by_admins_only) { ui.report_success("New users must be invited by an admin!", invite_by_admins_
˓→only_status); } else { ui.report_success("Any user may now invite new users!", invite_by_admins_only_
˓→status); } # ... } });
Finally, update server_events.js to handle related events coming from the server. :
# static/js/server_events.js function dispatch_normal_event(event) { switch (event.type) { # ... case'realm': if (event.op ==='update'&& event.property ==='invite_by_admins_only'){ page_params.realm_invite_by_admins_only= event.value; } } }
Any code needed to update the UI should be placed in dispatch_normal_event callback (rather than the channel.patch) function. This ensures the appropriate code will run even if the changes are made in another browser window. In this example most of the changes are on the backend, so no UI updates are required.
20.2. Example Feature 113 Zulip Documentation, Release 1.4.0
114 Chapter 20. Writing a new application feature CHAPTER 21
Writing views in Zulip
21.1 What this covers
This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide, and treat this as a reference. If you have experience with Django, much of this will be familiar, but you may want to read about how REST requests are dispatched, and how request authentication works. This document supplements the new feature tutorial and the testing documentation.
21.2 What is a view?
A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’t show a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations, which shows the integration documentation. These paths show up in the address bar of the browser. There are other views that are only seen by software, namely the API views. They are used to build the various clients that Zulip has, namely the web client (which is also used by the desktop client) and the mobile clients.
21.3 Modifying urls.py
A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serve HTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations). The format of the URL patterns in Django is documented here, and the Zulip specific details for these are discussed in detail in the life of a request doc. We have two Zulip-specific conventions we use for internationalization and for our REST API, respectively.
115 Zulip Documentation, Release 1.4.0
21.4 Writing human-readable views
If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py
i18n_urls = [ ... + url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html
˓→')), + url(r'^postcards/$', 'zerver.views.postcards'), ]
As an example, if a request comes in for Spanish, language code es, the server path will be something like: es/ features/.
21.4.1 Decorators used for webpage views
This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators. require_post:
@require_post def accounts_register(request): # type: (HttpRequest) -> HttpResponse
This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page is requested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll get a HTTP 405 METHOD NOT ALLOWED error. zulip_login_required: This decorator verifies that the browser is logged in (i.e. has a valid session cookie) before providing the view for this route, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If a request comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork of Django’s login_required, adding a few extra checks specific to Zulip.
@zulip_login_required def home(request): # type: (HttpRequest) -> HttpResponse
21.4.2 Writing a template
Templates for the main website are found in templates/zerver.
21.5 Writing API REST endpoints
These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-string responses. Almost all Zulip view code is in the implementations of API REST endpoints. The REST API does authentication of the user through rest_dispatch, which is documented in detail at zerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser, or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.
116 Chapter 21. Writing views in Zulip Zulip Documentation, Release 1.4.0
>>> import requests >>> r= requests.get('https://api.github.com/user', auth=('[email protected]',
˓→'0123456789abcdeFGHIJKLmnopQRSTUV')) >>> r.status_code -> 200
21.5.1 Request variables
Most API views will have some arguments that are passed as part of the request to control the behavior of the view. In any well-engineered view, you need to write code to parse and validate that the arguments exist and have the correct form. For many applications, this leads to one of several bad outcomes: • The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users of the API. • Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error messages. • Every view function comes with another function that does the validation that has the problems from the last bullet point. In Zulip, we solve this problem with a the special decorator called has_request_variables which allows a developer to declare the arguments a view function takes and validate their types all within the def line of the function. We like this framework because we have found it makes the validation code compact, readable, and conveniently located in the same place as the method it is validating arguments for. Here’s an example:
from zerver.decorator import has_request_variables, REQ, JsonableError, \ require_realm_admin
@require_realm_admin @has_request_variables def create_user_backend(request, user_profile, email=REQ(), password=REQ(), full_name=REQ(), short_name=REQ()): # ... code here
You will notice the special REQ() in the keyword arguments to create_user_backend. has_request_variables parses the declared keyword arguments of the decorated function, and for each that has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request, parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the event that an argument is missing, doesn’t parse as JSON, or otherwise is invalid. require_realm_admin is another decorator which checks the authorization of the given user_profile to make sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily to show how has_request_variables should be the inner decorator. The implementation of has_request_variables is documented in detail in zerver/lib/request.py) REQ also helps us with request variable validation. For example: • msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP pa- rameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keyword argument. • streams_raw = REQ("subscriptions", validator=check_list(check_string)) will check that the “subscriptions” HTTP parameter is a list of strings, marshalled as JSON, and pass it into the function with the Python keyword argument streams_raw.
21.5. Writing API REST endpoints 117 Zulip Documentation, Release 1.4.0
• message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTP parameter is a string containing a non-negative integer (converter differs from validator in that it does not automatically marshall the input from JSON). See zerver/lib/validator.py for more validators and their documentation.
21.5.2 Deciding which HTTP verb to use
When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read or write operation. If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a resource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and used by clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as it happens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more. If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’re creating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If you want to create the data multiple times for multiple requests (for example, requesting the send_message view multiple times with the same content should send multiple messages), you should use POST. If you’re updating existing data, use PATCH. If you’re removing data, use DELETE.
21.5.3 Idempotency
When writing a new API endpoint, with the exception of things like sending messages, requests should be safe to repeat, without impacting the state of the server. This is idempotency. You will often want to return an error if a request to change something would do nothing because the state is already as desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be the same, but the repeated requests won’t change the server more than once or cause unwanted side effects.
21.5.4 Making changes to the database
If the view does any modification to the database, that change is done in a helper function in zerver/lib/ actions.py. Those functions are responsible for doing a complete update to the state of the server, which often entails both updating the database and sending any events to notify clients about the state change. When possible, we prefer to design a clean boundary between the view function and the actions function is such that all user input validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for atomically executing the change (this is usually signalled by having the actions function have a name starting with do_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection to the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a name starting with check_. For example, in zerver/views/realm.py:
@require_realm_admin @has_request_variables def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None),...)): # type: (HttpRequest, UserProfile, ...) -> HttpResponse realm= user_profile.realm data={} # type: Dict[str, Any]
118 Chapter 21. Writing views in Zulip Zulip Documentation, Release 1.4.0
if name is not None and realm.name != name: do_set_realm_name(realm, name) data['name']='updated' and in zerver/lib/actions.py: def do_set_realm_name(realm, name): # type: (Realm, Text) -> None realm.name= name realm.save(update_fields=['name']) event= dict( type="realm", op="update", property='name', value=name, ) send_event(event, active_user_ids(realm)) realm.save() actually saves the changes to the realm to the database, and send_event sends the event to active clients belonging to the provided list of users (in this case, all altive users in the Zulip realm).
21.5.5 Calling from the web application
You should always use channel. to make an HTTP
21.5.6 Calling from an API client
Here’s how you might manually make a call from python: payload={'name': new_name}
# email and API key api_auth=('[email protected]','0123456789abcdeFGHIJKLmnopQRSTUV') r= requests.patch(SERVER_URL+'api/v1/realm', data=json.dumps(payload), auth=api_auth, )
This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a nice interface for accessing the API.
21.5. Writing API REST endpoints 119 Zulip Documentation, Release 1.4.0
21.6 Legacy endpoints used by the web client
New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage of a browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your feature available to other clients, especially the mobile clients. These endpoints make use of some older authentication decorators, authenticated_json_api_view, authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.
21.7 Webhook integration endpoints
Webhooks are called by other services, often to send a message as part of those services’ integrations. They are most often POST requests, and often there is very little you can customize about them. Usually you can expect that the webhook for a service will allow specification for the target server for the webhook, and an API key. If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator, to fill in the user_profile and client fields of a request:
@api_key_only_webhook_view('PagerDuty') @has_request_variables def api_pagerduty_webhook(request, user_profile, client, payload=REQ(argument_type='body'), stream=REQ(default='pagerduty'), topic=REQ(default=None)):
The client will be the result of get_client("ZulipPagerDutyWebhook") in this example.
120 Chapter 21. Writing views in Zulip CHAPTER 22
Life of a Request
It can sometimes be confusing to figure out how to write a new feature, or debug an existing one. Let us try to follow a request through the Zulip codebase, and dive deep into how each part works. We will use as our example the creation of users through the API, but we will also highlight how alternative requests are handled.
22.1 A request is sent to the server, and handled by Nginx
When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know how this works, except for when it isn’t working. Nginx does the first level of routing–deciding which application will serve the request (or deciding to serve the request itself for static content). In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static, and are served unminified.
22.2 Nginx secures traffic with SSL
If you visit your Zulip server in your browser and discover that your traffic isn’t being properly encrypted, an nginx misconfiguration is the likely culprit.
22.3 Static files are served directly by Nginx
Static files include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3). location/static/{ alias/home/zulip/prod-static/; error_page 404/static/html/404.html; }
121 Zulip Documentation, Release 1.4.0
22.4 Nginx routes other requests between tornado and django
All our connected clients hold open long-polling connections so that they can receive events (messages, presence notifications, and so on) in real-time. Events are served by Zulip’s tornado application. Nearly every other kind of request is served by the zerver Django application. Here is the relevant nginx routing configuration.
22.5 Django routes the request to a view in urls.py files
There are various urls.py files throughout the server codebase, which are covered in more detail in the directory structure doc. The main Zulip Django app is zerver. The routes are found in
zproject/urls.py zproject/legacy_urls.py
There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types of requests are handled, and focus on how the REST API handles our user creation example.
22.6 Views serving HTML are internationalized by server path
If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of the browser, and serve HTML. For example, the /hello page (preview here) gets translated in Chinese at zh-cn/hello/ (preview here). Note the zh-cn prefix–that url pattern gets added by i18n_patterns.
22.7 API endpoints use REST
Our example is a REST API endpoint. It’s a PUT to /users. With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-out endpoints, Zulip uses REST for its API. This means that we use: • POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb is appropriate. • PUT for creating something for which we have a unique ID. • DELETE for deleting something • PATCH for updating or editing attributes of something. • GET to get something (read-only) • HEAD to check the existence of something to GET, without getting it; useful to check a link without download- ing a potentially large link • OPTIONS (handled automatically, see more below)
122 Chapter 22. Life of a Request Zulip Documentation, Release 1.4.0
Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the request multiple times and get the same state on the server. You might get a different response after the first request, as we like to give our clients an error so they know that no new change was made by the extra requests. POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH is special–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible. This cookbook and tutorial can be helpful if you are new to REST web applications.
22.7.1 PUT is only for creating new things
If you’re used to using PUT to update or modify resources, you might find our convention a little strange. We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (like sending a message with the same content multiple times), and PATCH to modify resources. In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.
22.7.2 OPTIONS
The OPTIONS method will yield the allowed methods. This request: OPTIONS https://chat.zulip.org/api/v1/users yields a response with this HTTP header: Allow: PUT, GET We can see this reflected in zproject/urls.py:
url(r'^users$','zerver.lib.rest.rest_dispatch', {'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'}),
In this way, the API is partially self-documenting.
22.7.3 Legacy endpoints are used by the web client
The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the web client, and use POST. You can see them in zproject/legacy_urls.py.
22.7.4 Webhook integrations may not be RESTful
Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. They are likely to use only POST.
22.8 Django calls rest_dispatch for REST endpoints, and authenti- cates
For requests that correspond to a REST url pattern, Zulip configures its url patterns (see zerver/lib/rest.py) so that the action called is rest_dispatch. This method will authenticate the user, either through a session token from a cookie, or from an email:api-key string given via HTTP Basic Auth for API clients.
22.8. Django calls rest_dispatch for REST endpoints, and authenticates 123 Zulip Documentation, Release 1.4.0
It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then figure out which view to show from that. In our example,
{'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'} is supplied as an argument to rest_dispatch, along with the HTTPRequest. The request has the HTTP verb PUT, which rest_dispatch can use to find the correct view to show: zerver.views.users. create_user_backend.
22.9 The view will authorize the user, extract request variables, and validate them
This is covered in good detail in the writing views doc.
22.10 Results are given as JSON
Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of an error, which gives a JSON string: {'result': 'error', 'msg':
124 Chapter 22. Life of a Request CHAPTER 23
Reading list
Here you can find a list of relevant resources that you may find useful for learning new skills, or polishing the ones you already have. The topics cover a wide variety of topics, from basic Python coding to general developing guidelines. Feel free to create a pull request in Zulip’s GitHub repository with any interesting books, articles or videos you would like to see in this list. Some titles have been shortened for organizational purposes.
23.1 General programming/IT
Book - Clean Code: A Handbook of Agile Software Craftsmanship (Not free!) Books - Free programming books list Blog - Free Code Camp blog Blog - Idle Words talks transcripts Tutorial - HTTP Can Do That?!, by Sumana Harihareswara (PyCon 2016) Video - Minimum Viable Documentation, by Matthew Lyon (WriteTheDocs 2014) Video - NoOps, by Kelsey Hightower (DepOpsDays 2016) Video - The mind behind Linux (TED interview) Tutorial - Learn code the hard way Tutorial - What happens when... Article - An Interview With Linus Torvalds Article - Effective Learning Strategies for Programmers Article - Readme Driven Development
125 Zulip Documentation, Release 1.4.0
Article - Systematic Debugging Paper - Floating-Point Arithmetic
23.2 Python
Video - Intro to Python for beginners, by Jessica McKellar (PyCon 2013) Video - Breaking the rules, by Jessica McKellar (PyCon Sweden) Video - Build & break a Python sandbox, by Jessica McKellar (PyCon 2014) Video - Cache me if you can, by Guillaume Ardaud (PyCon 2014) Video - Loop like a native, by Ned Batchelder (PyCon 2013) Video - Modern Dictionaries, by Raymond Hettinger (SF Python) Video - Python Language, by Guido van Rossum (PyCon 2016) Video - The Mighty Dictionary, by Brandon Rhodes (PyCon 2010) Tutorial - Code Like a Pythonista: Idiomatic Python Article - Static types in Python, oh my(py)! Guide - The Hitchhiker’s Guide to Python!
23.3 Java/Android
Course - Android Development for Beginners
23.4 JavaScript/ECMAScript
Tutorial - clean-code-javascript Software engineering principles Slides - TypeScript vs. CoffeeScript vs. ES6
23.5 Git/Version Control Systems (VCS)
You may want to take a look first at our Git and GitHub guide. Article - Git tips
23.6 Computer Science/Algorithms
Blog - GeeksforGeeks Book Introduction to Algorithms( Not free!) Blog - Setosa data visualization and visual explanations Course - Algorithms, Part I
126 Chapter 23. Reading list Zulip Documentation, Release 1.4.0
Course - Open Source Society University Course - MIT CSAIL 6.828: Operative Systems Engineering
23.7 Community experience
Book - Producing Open Source Software Article - Advice on Starting And Running A New Open Source Project Article - How to ask good questions Article - Notes for New FLOSS Contributors Article - To be mentored Article - To mentor List of good projects for new contributors
23.8 Competitions/Camps
CodeForces Free Code Camp
23.9 Massive Open Online Courses (MOOC) Platforms
Coursera edX MIT OpenCourseWare Udacity
23.7. Community experience 127 Zulip Documentation, Release 1.4.0
128 Chapter 23. Reading list CHAPTER 24
Git & GitHub Guide
24.1 Quick start: How Zulip uses Git and GitHub
This quick start provides a brief overview of how Zulip uses Git and GitHub. Those who are familiar with Git and GitHub should be able to start contributing with these details in mind: • We use GitHub for source control and code review. To contribute, fork zulip/zulip (or the appropriate repos- itory, if you are working on something else besides Zulip server) to your own account and then create fea- ture/issue branches. When you’re ready to get feedback, submit a work-in-progress (WIP) pull request. We encourage you to submit WIP pull requests early and often. • We use a rebase-oriented workflow. We do not use merge commits. This means you should use git fetch followed by git rebase rather than git pull (or you can use git pull --rebase). Also, to prevent pull requests from becoming out of date with the main line of development, you should rebase your feature branch prior to submitting a pull request, and as needed thereafter. If you’re unfamiliar with how to rebase a pull request, read this excellent guide. We use this strategy in order to avoid the extra commits that appear when another branch is merged, that clutter the commit history (it’s popular with other large projects such as Django). This makes Zulip’s commit history more readable, but a side effect is that many pull requests we merge will be reported by GitHub’s UI as closed instead of merged, since GitHub has poor support for rebase-oriented workflows. • We have a code style guide, a commit message guide, and strive for each commit to be a minimal coherent idea (see commit discipline for details). • We provide many tools to help you submit quality code. These include linters, tests, continuous integration with TravisCI, and mypy. Finally, take a quick look at Zulip-specific Git scripts, install the Zulip developer environment, and then configure your fork for use with TravisCI.
The following sections will help you be awesome with Zulip and Git/GitHub in a rebased-based workflow. Read through it if you’re new to git, to a rebase-based git workflow, or if you’d like a git refresher.
129 Zulip Documentation, Release 1.4.0
24.2 Set up Git
If you’re already using Git, have a client you like, and a GitHub account, you can skip this section. Otherwise, read on!
24.2.1 Install and configure Git, join GitHub
If you’re not already using Git, you might need to install and configure it. You’ll also need a GitHub account, which you can sign up for here. We also recommend you create an ssh key if you don’t already have one and add it to your GitHub account. If you plan on using Git from the command line, the following tips can make your experience better: • Install the command auto-completion and/or git-prompt plugins available for Bash and Zsh. • Configure Git with your user details and aliases for commands you’ll use often.
24.2.2 Get a graphical client
Even if you’re comfortable using git on the command line, having a graphic client can be useful for viewing your repository. This is especially when doing a complicated rebases and similar operations because you can check the state of your repository after each command to see what changed. If something goes wrong, this helps you figure out when and why. If you don’t already have one installed, here are some suggestions: • macOS: GitX-dev • Ubuntu/Linux: git-cola, gitg, gitk • Windows: SourceTree If you like working on the command line, but want better visualization and navigation of your git repo, try Tig, a cross-platform ncurses-based text-mode interface to Git. And, if none of the above are to your liking, try one of these.
24.3 How Git is different
Whether you’re new to Git or have experience with another version control system (VCS), it’s a good idea to learn a bit about how Git works. We recommend this excellent presentation Understanding Git from Nelson Elhage and Anders Kaseorg and the Git Basics chapter from Pro Git by Scott Chacon and Ben Straub. Here are the top things to know: • Git works on snapshots: Unlike other version control systems (e.g., Subversion, Perforce, Bazaar), which track files and changes to those files made over time, Git tracks snapshots of your project. Each time you commit or otherwise make a change to your repository, Git takes a snapshot of your project and stores a reference to that snapshot. If a file hasn’t changed, Git creates a link to the identical file rather than storing it again. • Most Git operations are local: Git is a distributed version control system, so once you’ve cloned a repository, you have a complete copy of that repository’s entire history. Staging, committing, branching, and browsing history are all things you can do locally without network access and without immediately affecting any remote repositories. To make or receive changes from remote repositories, you need to git fetch, git pull, or git push.
130 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
• Nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, Git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository. • Git is designed for lightweight branching and merging. Branches are simply references to snapshots. It’s okay and expected to make a lot of branches, even throwaway and experimental ones. • Git stores all data as objects, of which there are four types: blob (file), tree (directory), commit (revision), and tag. Each of these objects is named by a unique hash, the SHA-1 has of its contents. Most of the time you’ll refer to objects by their truncated hash or more human-readable reference like HEAD (the current branch). Blobs and trees represent files and directories. Tags are named references to other objects. A commit object includes: tree id, zero or more parents as commit ids, an author (name, email, date), a committer (name, email, date), and a log message. A Git repository is a collection of mutable pointers to these objects called refs. • Cloning a repository creates a working copy. Every working copy has a .git subdirectory, which contains its own Git repository. The .git subdirectory also tracks the index, a staging area for changes that will become part of the next commit. All files outside of .git is the working tree. • Files tracked with Git have possible three states: committed, modified, and staged. Committed files are those safely stored in your local .git repository/database. Staged files have changes and have been marked for inclusion in the next commit; they are part of the index. Modified files have changes but have not yet been marked for inclusion in the next commit; they have not been added to the index. • Git commit workflow is as follows: Edit files in your working tree. Add to the index (that is stage) with git add. Commit to the HEAD of the current branch with git commit.
24.4 Important Git terms
When you install Git, it adds a manual entry for gitglossary. You can view this glossary by running man gitglossary. Below we’ve included the git terms you’ll encounter most often along with their definitions from gitglossary.
24.4.1 branch
A “branch” is an active line of development. The most recent commit on a branch is referred to as the tip of that branch. The tip of the branch is referenced by a branch head, which moves forward as additional development is done on the branch. A single Git repository can track an arbitrary number of branches, but your working tree is associated with just one of them (the “current” or “checked out” branch), and HEAD points to that branch.
24.4.2 cache
Obsolete for: index
24.4.3 checkout
The action of updating all or part of the working tree with a tree object or blob from the object database, and updating the index and HEAD if the whole working tree has been pointed at a new branch.
24.4. Important Git terms 131 Zulip Documentation, Release 1.4.0
24.4.4 commit
As a noun: A single point in the Git history; the entire history of a project is represented as a set of interrelated commits. The word “commit” is often used by Git in the same places other revision control systems use the words “revision” or “version”. Also used as a short hand for commit object. As a verb: The action of storing a new snapshot of the project’s state in the Git history, by creating a new commit representing the current state of the index and advancing HEAD to point at the new
24.4.5 fast-forward
A fast-forward is a special type of merge where you have a revision and you are “merging” another branch’s changes that happen to be a descendant of what you have. In such these cases, you do not make a new mergecommit but instead just update to his revision. This will happen frequently on a remote-tracking branch of a remote repository.
24.4.6 fetch
Fetching a branch means to get the branch’s head ref from a remote repository, to find out which objects are missing from the local object database, and to get them, too. See also git-fetch(1).
24.4.7 hash
In Git’s context, synonym for object name.
24.4.8 head
A named reference to the commit at the tip of a branch. Heads are stored in a file in $GIT_DIR/refs/heads/ directory, except when using packed refs. (See git-pack-refs(1).)
24.4.9 HEAD
The current branch. In more detail: Your working tree is normally derived from the state of the tree referred to by HEAD. HEAD is a reference to one of the heads in your repository, except when using a detached HEAD, in which case it directly references an arbitrary commit.
24.4.10 index
A collection of files with stat information, whose contents are stored as objects. The index is a stored version of your working tree. Truth be told, it can also contain a second, and even a third version of a working tree, which are used when merging.
24.4.11 pull
Pulling a branch means to fetch it and merge it. See also git- pull(1).
132 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
24.4.12 push
Pushing a branch means to get the branch’s head ref from a remote repository, find out if it is a direct ancestor to the branch’s local head ref, and in that case, putting all objects, which are reachable from the local head ref, and which are missing from the remote repository, into the remote object database, and updating the remote head ref. If the remote head is not an ancestor to the local head, the push fails.
24.4.13 rebase
To reapply a series of changes from a branch to a different base, and reset the head of that branch to the result.
24.5 Get Zulip code
Zulip uses a forked-repo and rebase-oriented workflow.. This means that all contributors create a fork of the Zulip repository they want to contribute to and then submit pull requests to the upstream repository to have their contributions reviewed and accepted. We also recommend you work on feature branches.
24.5.1 Step 1a: Create your fork
The following steps you’ll only need to do the first time you setup a machine for contributing to a given Zulip project. You’ll need to repeat the steps for any additional Zulip projects (list) that you work on. The first thing you’ll want to do to contribute to Zulip is fork (see how) the appropriate Zulip repository. For the main server app, this is zulip/zulip.
24.5.2 Step 1b: Clone to your machine
Next, clone your fork to your local machine:
$ git clone [email protected]:christi3k/zulip.git Cloning into 'zulip' remote: Counting objects: 86768, done. remote: Compressing objects: 100% (15/15), done. remote: Total 86768 (delta 5), reused 1 (delta 1), pack-reused 86752 Receiving objects: 100% (86768/86768), 112.96 MiB | 523.00 KiB/s, done. Resolving deltas: 100% (61106/61106), done. Checking connectivity... done.
Note: If you receive an error while cloning, you may not have added your ssh key to GitHub.
24.5.3 Step 1c: Connect your fork to Zulip upstream
Next you’ll want to configure an upstream remote repository for your fork of Zulip. This will allow you to sync changes from the main project back into your fork. First, show the currently configured remote repository:
$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push)
24.5. Get Zulip code 133 Zulip Documentation, Release 1.4.0
Note: If you’ve cloned the repository using a graphical client, you may already have the upstream remote repository configured. For example, when you clone zulip/zulip with the GitHub desktop client it configures the remote repository zulip and you see the following output from git remote -v: origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) zulip https://github.com/zulip/zulip.git (fetch) zulip https://github.com/zulip/zulip.git (push)
If your client hasn’t automatically configured a remote for zulip/zulip, you’ll need to with:
$ git remote add upstream https://github.com/zulip/zulip.git
Finally, confirm that the new remote repository, upstream, has been configured:
$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) upstream https://github.com/zulip/zulip.git (fetch) upstream https://github.com/zulip/zulip.git (push)
24.5.4 Step 2: Set up the Zulip development environment
If you haven’t already, now is a good time to install the Zulip development environment (overview). If you’re new to working on Zulip or open source projects in general, we recommend following our detailed guide for first-time contributors.
24.5.5 Step 3: Configure Travis CI (continuous integration)
This step is optional, but recommended. Zulip Server is configured to use Travis CI to test and create builds upon each new commit and pull request. Travis CI is free for open source projects and it’s easy to configure for your own fork of Zulip. After doing so, TravisCI will run tests for new refs you push to GitHub and email you the outcome (you can also view the results in the web interface). First, sign in to Travis CI with your GitHub account and authorize Travis CI to access your GitHub account and repositories. Once you’ve done this, Travis CI will fetch your repository information and display it on your profile page. From there you can enable integration with Zulip. (See screen cast.)
24.6 Using Git as you work
24.6.1 Know what branch you’re working on
When using Git, it’s important to know which branch you currently have checked out because most git commands implicitly operate on the current branch. You can determine the currently checked out branch several ways. One way is with git status:
$ git status On branch issue-demo nothing to commit, working directory clean
Another is with git branch which will display all local branches, with a star next to the current branch:
134 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
$ git branch * issue-demo master
To see even more information about your branches, including remote branches, use git branch -vva:
$ git branch -vva * issue-123 517468b troubleshooting tip about provisioning master f0eaee6 [origin/master] bug: Fix traceback in get_missed_
˓→message_token_from_address(). remotes/origin/HEAD -> origin/master remotes/origin/issue-1234 4aeccb7 Another test commit, with longer message. remotes/origin/master f0eaee6 bug: Fix traceback in get_missed_message_token_
˓→from_address(). remotes/upstream/master dbeab6a Optimize checks of test database state by moving
˓→into Python.
You can also configure Bash and Zsh to display the current branch in your prompt.
24.6.2 Keep your fork up to date
You’ll want to keep your fork up-to-date with changes from Zulip’s main repositories. Note about git pull: You might be used to using git pull on other projects. With Zulip, because we don’t use merge commits, you’ll want to avoid it. Rather that using git pull, which by default is a shortcut for git fetch && git merge FETCH_HEAD (docs), you should use git fetch and then git rebase. First, fetch changes from Zulip’s upstream repository you configured in the step above:
$ git fetch upstream
Next, checkout your master branch and rebase it on top of upstream/master:
$ git checkout master Switched to branch 'master'
$ git rebase upstream/master
This will rollback any changes you’ve made to master, update it from upstream/master, and then re-apply your changes. Rebasing keeps the commit history clean and readable. When you’re ready, push your changes to your remote fork. Make sure you’re in branch master and the run git push:
$ git checkout master $ git push origin master
You can keep any branch up to date using this method. If you’re working on a feature branch (see next section), which we recommend, you would change the command slightly, using the name of your feature-branch rather than master:
$ git checkout feature-branch Switched to branch 'feature-branch'
$ git rebase upstream/master
$ git push origin feature-branch
24.6. Using Git as you work 135 Zulip Documentation, Release 1.4.0
24.6.3 Work on a feature branch
One way to keep your work organized is to create a branch for each issue or feature. Recall from how Git is different that Git is designed for lightweight branching and merging. You can and should create as many branches as you’d like. First, make sure your master branch is up-to-date with Zulip upstream (see how). Next, from your master branch, create a new tracking branch, providing a descriptive name for your feature branch:
$ git checkout master Switched to branch 'master'
$ git checkout -b issue-1755-fail2ban Switched to a new branch 'issue-1755-fail2ban'
Alternatively, you can create a new branch explicitly based off upstream/master:
$ git checkout -b issue-1755-fail2ban upstream/master Switched to a new branch 'issue-1755-fail2ban'
Now you’re ready to work on the issue or feature.
24.6.4 Run linters and tests locally
In addition to having Travis run tests and linters each time you push a new commit, you can also run them locally. See testing for details.
24.6.5 Stage changes
Recall that files tracked with Git have possible three states: committed, modified, and staged. To prepare a commit, first add the files with changes that you want to include in your commit to your staging area. You add both new files and existing ones. You can also remove files from staging when necessary.
Get status of working directory
To see what files in the working directory have changes that have not been staged, use git status. If you have no changes in the working directory, you’ll see something like this:
$ git status On branch issue-123 nothing to commit, working directory clean
If you have unstaged changes, you’ll see something like this:
On branch issue-123 Untracked files: (use"git add
newfile.py
nothing added to commit but untracked files present (use"git add" to track)
136 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
Stage additions with git add
To add changes to your staging area, use git add
On branch issue-123 Changes to be committed: (use"git reset HEAD
new file: newfile.py
You can view the changes in files you have staged with git diff --cached. To view changes to files you haven’t yet staged, just use git diff. If you want to add all changes in the working directory, use git add -A (documentation). You can also stage changes using your graphical Git client. If you stage a file, you can undo it with git reset HEAD
$ git add test3.txt On branch issue-1234 Changes to be committed: (use "git reset HEAD
new file: test3.txt
$ git reset HEAD test3.txt $ git status On branch issue-1234 Untracked files: (use "git add
test3.txt
nothing added to commit but untracked files present (use "git add" to track)
Stage deletions with git rm
To remove existing files from your repository, use git rm (documentation). This command can either stage the file for removal from your repository AND delete it from your working directory or just stage the file for deletion and leave it in your working directory. To stage a file for deletion and remove it from your working directory, use git rm
$ git rm test.txt rm 'test.txt'
$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD
deleted: test.txt
24.6. Using Git as you work 137 Zulip Documentation, Release 1.4.0
$ ls test.txt ls: No such file or directory
To stage a file for deletion and keep it in your working directory, use git rm --cached
$ git rm --cached test2.txt rm 'test2.txt'
$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD
deleted: test2.txt
$ ls test2.txt test2.txt
If you stage a file for deletion with the --cached option, and haven’t yet run git commit, you can undo it with git reset HEAD
$ git reset HEAD test2.txt
Unfortunately, you can’t restore a file deleted with git rm if you didn’t use the --cache option. However, git rm only deletes files it knows about. Files you have never added to git won’t be deleted.
24.6.6 Commit changes
When you’ve staged all your changes, you’re ready to commit. You can do this with git commit -m "My commit message." to include a commit message. Here’s an example of committing with the -m for a one-line commit message:
$ git commit -m "Add a test commit for docs." [issue-123 173e17a] Add a test commit for docs. 1 file changed, 1 insertion(+) create mode 100644 newfile.py
You can also use git commit without the -m option and your editor to open, allowing you to easily draft a multi-line commit message. How long your commit message should be depends on where you are in your work. Using short, one-line messages for commits related to in-progress work makes sense. For a commit that you intend to be final or that encompasses a significant amount or complex work, you should include a longer message. Keep in mind that your commit should contain a ‘minimal coherent idea’ and have a quality commit message. See Zulip docs Commit Discipline and Commit messages for details. Here’s an example of a longer commit message that will be used for a pull request:
Integrate Fail2Ban.
Updates Zulip logging to put an unambiguous entry into the logs such that fail2ban can be configured to look for these entries.
Tested on my local Ubuntu development server, but would appreciate
138 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
someone testing on a production install with more users.
Fixes #1755.
The first line is the summary. It’s a complete sentence, ending in a period. It uses a present-tense action verb, “Integrate”, rather than “Integrates” or “Integrating”. The following paragraphs are full prose and explain why and how the change was made. It explains what testing was done and asks specifically for further testing in a more production-like environment. The final paragraph indicates that this commit addresses and fixes issue #1755. When you submit your pull request, GitHub will detect and link this reference to the appropriate issue. Once your commit is merged into zulip/master, GitHub will automatically close the referenced issue. See Closing issues via commit messages for details. Make as many commits as you need to to address the issue or implement your feature.
24.6.7 Push your commits to GitHub
As you’re working, it’s a good idea to frequently push your changes to GitHub. This ensures your work is backed up should something happen to your local machine and allows others to follow your progress. It also allows you to work from multiple computers without losing work. Pushing to a feature branch is just like pushing to master:
$ git push origin
If you want to see what git will do without actually performing the push, add the -n (dry-run) option: git push -n origin
24.6.8 Examine and tidy your commit history
Examining your commit history prior to submitting your pull request is a good idea. Is it tidy such that each commit represents a minimally coherent idea (see commit discipline)? Do your commit messages follow Zulip’s style? Will the person reviewing your commit history be able to clearly understand your progression of work? On the command line, you can use the git log command to display an easy to read list of your commits:
$ git log --all --graph --oneline --decorate
* 4f8d75d (HEAD -> 1754-docs-add-git-workflow) docs: Add details about configuring ˓→Travis CI. * bfb2433 (origin/1754-docs-add-git-workflow) docs: Add section for keeping fork up- ˓→to-date to Git Guide. * 4fe10f8 docs: Add sections for creating and configuring fork to Git Guide. * 985116b docs: Add graphic client recs to Git Guide. * 3c40103 docs: Add stubs for remaining Git Guide sections.
24.6. Using Git as you work 139 Zulip Documentation, Release 1.4.0
* fc2c01e docs: Add git guide quickstart. | * f0eaee6 (upstream/master) bug: Fix traceback in get_missed_message_token_from_ ˓→address().
Alternatively, use your graphical client to view the history for your feature branch. If you need to update any of your commits, you can do so with an interactive rebase. Common reasons to use an interactive rebase include: • squashing several commits into fewer commits • splitting a single commit into two or more • rewriting one or more commit messages There is ample documentation on how to rebase, so we won’t go into details here. We recommend starting with GitHub’s help article on rebasing and then consulting Git’s documentation for git-rebase if you need more details. If all you need to do is edit the commit message for your last commit, you can do that with git commit --amend. See Git Basics - Undoing Things for details on this and other useful commands.
24.6.9 Force-push changes to GitHub after you’ve altered your history
Any time you alter history for commits you have already pushed to GitHub, you’ll need to prefix the name of your branch with a +. Without this, your updates will be rejected with a message such as:
$ git push origin 1754-docs-add-git-workflow To [email protected]:christi3k/zulip.git ! [rejected] 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (non-fast-
˓→forward) error: failed to push some refs to '[email protected]:christi3k/zulip.git' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Re-running the command with +
$ git push origin +1754-docs-add-git-workflow Counting objects: 12, done. Delta compression using up to 4 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 3.71 KiB | 0 bytes/s, done. Total 12 (delta 8), reused 0 (delta 0) remote: Resolving deltas: 100% (8/8), completed with 2 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (forced
˓→update)
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the branch. If others are working along with you, they might run into complications when they retrieve your changes because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
140 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
24.7 Create a pull request
When you’re ready for feedback, submit a pull request. At Zulip we recommend submitting work-in-progress pull requests early and often. This allows you to get feedback and help with your bugfix or feature. Prefix work-in-progress pull requests with [WIP]. Pull requests are a feature specific to GitHub. They provide a simpler, web-based way to submit your work (often called “patches”) to a project. It’s called a pull request because you’re asking the project to pull changes from your fork. If you’re unfamiliar with how to create a pull request, checkout GitHub’s documentation on creating a pull request from a fork. You might also find GitHub’s article about pull requests helpful.
24.7.1 Step 1: Update your branch with git rebase
The best way to update your branch is with git fetch and git rebase. Do not use git pull or git merge as this will create merge commits. See keep your fork up to date for details. Here’s an example (you would replace issue-123 with the name of your feature branch):
$ git checkout issue123 Switched to branch 'issue-123'
$ git fetch upstream remote: Counting objects: 69, done. remote: Compressing objects: 100% (23/23), done. remote: Total 69 (delta 49), reused 39 (delta 39), pack-reused 7 Unpacking objects: 100% (69/69), done. From https://github.com/zulip/zulip 69fa600..43e21f6 master -> upstream/master
$ git rebase upstream/master
First, rewinding head to replay your work on top of it... Applying: troubleshooting tip about provisioning
24.7.2 Step 2: Push your updated branch to your remote fork
Once you’ve updated your local feature branch, push the changes to GitHub:
$ git push origin issue-123 Counting objects: 6, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123
If your push is rejected with error failed to push some refs then you need to prefix the name of your branch with a +:
$ git push origin +issue-123 Counting objects: 6, done. Delta compression using up to 4 threads.
24.7. Create a pull request 141 Zulip Documentation, Release 1.4.0
Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123 (forced update)
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the branch. If others are working along with you, they might run into complications when they retrieve your changes because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
24.7.3 Step 3: Open the pull request
If you’ve never created a pull request or need a refresher, take a look at GitHub’s article creating a pull request from a fork. We’ll briefly review the process here. The first step in creating a pull request is to use your web browser to navigate to your fork of Zulip. Sign in to GitHub if you haven’t already. Next, navigate to the branch you’ve been working on. Do this by clicking on the Branch button and selecting the relevant branch. Finally, click the New pull request button. Alternatively, if you’ve recently pushed to your fork, you will see a green Compare & pull request button. You’ll see the Open a pull request page:
142 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
Provide a title and first comment for your pull request. When ready, click the green Create pull request to submit the pull request. Note: Pull request titles are different from commit messages. Commit messages can be edited with git commit --amend, git rebase -i, etc., while the title of a pull request can only be edited via GitHub.
24.8 Update a pull request
As you get make progress on your feature or bugfix, your pull request, once submitted, will be updated each time you push commits to your remote branch. This means you can keep your pull request open as long as you need, rather than closing and opening new ones for the same feature or bugfix. It’s a good idea to keep your pull request mergeable with Zulip upstream by frequently fetching, rebasing, and pushing changes. See keep your fork up to date for details. You might also find this excellent article How to Rebase a Pull Request helpful. And, as you address review comments others have made, we recommend posting a follow-up comment in which you: a) ask for any clarifications you need, b) explain to the reviewer how you solved any problems they mentioned, and c) ask for another review.
24.9 Collaborate
24.9.1 Fetch another contributor’s branch
What happens when you would like to collaborate with another contributor and they have work-in-progress on their own fork of Zulip? No problem! Just add their fork as a remote and pull their changes.
$ git remote add
Now you can checkout their branch just like you would any other. You can name the branch anything you want, but using both the username and branch name will help you keep things organized.
$ git checkout -b
24.9.2 Checkout a pull request locally
Just as you can checkout any user’s branch locally, you can also checkout any pull request locally. GitHub provides a special syntax (details) for this since pull requests are specific to GitHub rather than Git. First, fetch and create a branch for the pull request, replacing ID and BRANCHNAME with the ID of the pull request and your desired branch name:
$ git fetch upstream pull/ID/head:BRANCHNAME
Now switch to the branch:
$ git checkout BRANCHNAME
Now you work on this branch as you would any other.
24.8. Update a pull request 143 Zulip Documentation, Release 1.4.0
24.10 Review changes
24.10.1 Changes on (local) working tree
Display changes between index and working tree (what is not yet staged for commit):
$ git diff
Display changes between index and last commit (what you have staged for commit):
$ git diff --cached
Display changes in working tree since last commit (changes that are staged as well as ones that are not):
$ git diff HEAD
24.10.2 Changes within branches
Use any git-ref to compare changes between two commits on the current branch. Display changes between commit before last and last commit:
$ git diff HEAD^ HEAD
Display changes between two commits using their hashes:
$ git diff e2f404c 7977169
24.10.3 Changes between branches
Display changes between tip of topic branch and tip of master branch:
$ git diff topic master
Display changes that have occurred on master branch since topic branch was created:
$ git diff topic...master
Display changes you’ve committed so far since creating a branch from upstream/master:
$ git diff upstream/master...HEAD
24.11 Get and stay out of trouble
Git is a powerful yet complex version control system. Even for contributors experienced at using version control, it can be confusing. The good news is that nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository.
144 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
24.11.1 Undo a merge commit
A merge commit is a special type of commit that has two parent commits. It’s created by Git when you merge one branch into another and the last commit on your current branch is not a direct ancestor of the branch you are trying to merge in. This happens quite often in a busy project like Zulip where there are many contributors because upstream/zulip will have new commits while you’re working on a feature or bugfix. In order for Git to merge your changes and the changes that have occurred on zulip/upstream since you first started your work, it must perform a three-way merge and create a merge commit. Merge commits aren’t bad, however, Zulip don’t use them. Instead Zulip uses a forked-repo, rebase-oriented workflow. A merge commit is usually created when you’ve run git pull or git merge. You’ll know you’re creating a merge commit if you’re prompted for a commit message and the default is something like this:
Merge branch'master' of https://github.com/zulip/zulip
# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.
And the first entry for git log will show something like: commit e5f8211a565a5a5448b93e98ed56415255546f94 Merge: 13bea0e e0c10ed Author: Christie Koehler
Merge branch'master' of https://github.com/zulip/zulip
Some graphical Git clients may also create merge commits. To undo a merge commit, first run git reflog to identify the commit you want to roll back to:
$ git reflog e5f8211 HEAD@{0}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{1}: commit: test commit for docs.
Reflog output will be long. The most recent git refs will be listed at the top. In the example above e5f8211 HEAD@{0}: is the merge commit made automatically by git pull and 13bea0e HEAD@{1}: is the last com- mit I made before running git pull, the commit that I want to rollback to. Once you’d identified the ref you want to revert to, you can do so with git reset:
$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.
Important: git reset --hard
24.11. Get and stay out of trouble 145 Zulip Documentation, Release 1.4.0
$ git reflog
13bea0e HEAD@{2}: reset: moving to HEAD@{1} e5f8211 HEAD@{3}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{4}: commit: test commit for docs.
And the first entry git log shows is this: commit 13bea0e40197b1670e927a9eb05aaf50df9e8277 Author: Christie Koehler
test commit for docs.
24.11.2 Restore a lost commit
We’ve mentioned you can use git reset --hard to rollback to a previous commit. What if you run git reset --hard and then realize you actually need one or more of the commits you just discarded? No problem, you can restore them with git cherry-pick (docs). For example, let’s say you just committed “some work” and your git log looks like this:
* 67aea58 (HEAD-> master) some work * 13bea0e test commit for docs.
You then mistakenly run git reset --hard 13bea0e:
$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.
$ git log * 13bea0e (HEAD -> master) test commit for docs.
And then realize you actually needed to keep commit 67aea58. First, use git reflog to confirm that commit you want to restore and then run git cherry-pick
$ git reflog 13bea0e HEAD@{0}: reset: moving to 13bea0e 67aea58 HEAD@{1}: commit: some work
$ git cherry-pick 67aea58 [master 67aea58] some work Date: Thu Oct 13 11:51:19 2016 -0700 1 file changed, 1 insertion(+) create mode 100644 test4.txt
24.11.3 Recover from a git rebase failure
One situation in which git rebase will fail and require you to intervene is when your change, which git will try to re-apply on top of new commits from which ever branch you are rebasing on top of, is to code that has been changed by those new commits. For example, while I’m working on a file, another contributor makes a change to that file, submits a pull request and has their code merged into master. Usually this is not a problem, but in this case the other contributor made a change
146 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0 to a part of the file I also want to change. When I try to bring my branch up to date with git fetch and then git rebase upstream/master, I see the following:
First, rewinding head to replay your work on top of it... Applying: test change for docs Using index info to reconstruct a base tree... M README.md Falling back to patching base and 3-way merge... Auto-merging README.md CONFLICT (content): Merge conflict in README.md error: Failed to merge in the changes. Patch failed at 0001 test change for docs The copy of the patch that failed is found in:.git/rebase-apply/patch
When you have resolved this problem, run"git rebase --continue". If you prefer to skip this patch, run"git rebase --skip" instead. To check out the original branch and stop rebasing, run"git rebase --abort".
This message tells me that Git was not able to apply my changes to README.md after bringing in the new commits from upstream/master. Running git status also gives me some information: rebase in progress; onto5ae56e6 You are currently rebasing branch'docs-test' on'5ae56e6'. (fix conflicts and then run"git rebase --continue") (use"git rebase --skip" to skip this patch) (use"git rebase --abort" to check out the original branch)
Unmerged paths: (use"git reset HEAD
both modified: README.md no changes added to commit (use"git add" and/or "git commit -a")
To fix, open all the files with conflicts in your editor and decide which edits should be applied. Git uses standard conflict-resolution (<<<<<<<, ======, and >>>>>>>) markers to indicate where in files there are conflicts. Once you’ve done that, save the file(s), stage them with git add and then continue the rebase with git rebase --continue:
$ git add README.md
$ git rebase --continue Applying: test change for docs
For help resolving merge conflicts, see basic merge conflicts, advanced merging, and/or GitHub’s help on how to resolve a merge conflict.
24.11.4 Working from multiple computers
Working from multiple computers with Zulip and Git is fine, but you’ll need to pay attention and do a bit of work to ensure all of your work is readily available. Recall that most Git operations are local. When you commit your changes with git commit they are safely stored in your local Git database only. That is, until you push the commits to GitHub, they are only available on the computer
24.11. Get and stay out of trouble 147 Zulip Documentation, Release 1.4.0 where you committed them. So, before you stop working for the day, or before you switch computers, push all of your commits to GitHub with git push:
$ git push origin
When you first start working on a new computer, you’ll clone the Zulip repository and connect it to Zulip upstream. A clone retrieves all current commits, including the ones you pushed to GitHub from your other computer. But if you’re switching to another computer on which you have already cloned Zulip, you need to update your local Git database with new refs from your GitHub fork. You do this with git fetch:
$ git fetch
Ideally you should do this before you have made any commits on the same branch on the second computer. Then you can git merge on whichever branch you need to update:
$ git checkout
$ git merge origin/master
If you have already made commits on the second computer that you need to keep, you’ll need to use git log FETCH_HEAD to identify that hashes of the commits you want to keep and then git cherry-pick
24.12 Zulip-specific tools
This section will document the zulip-specific git tools contributors will find helpful.
24.12.1 Set up git repo script
In the tools directory of zulip/zulip you’ll find a bash script setup-git-repo. This script installs the Zulip pre-commit hook. This hook will run each time you git commit to automatically run linters, etc. The hook passes no matter the result of the linter, but you should still pay attention to any notices or warnings it displays. It’s simple to use. Make sure you’re in the clone of zulip and run the following:
$ ./tools/setup-git-repo
The script doesn’t produce any output if successful. To check that the hook has been installed, print a directory listing for .git/hooks and you should see something similar to:
$ ls -l .git/hooks pre-commit -> ../../tools/pre-commit
24.12.2 Reset to pull request tools/reset-to-pull-request is a short-cut for checking out a pull request locally. It works slightly differ- ently from the method described above in that it does not create a branch for the pull request checkout. You should run this script only while in the master branch. This script will discard any uncommitted changes. Use with caution.
148 Chapter 24. Git & GitHub Guide Zulip Documentation, Release 1.4.0
First, make sure you are working in branch master. Then run the script with the ID number of the pull request as the first argument.
$ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'.
$ ./tools/reset-to-pull-request 1900 + request_id=1900 + git fetch upstream pull/1900/head remote: Counting objects: 159, done. remote: Compressing objects: 100% (17/17), done. remote: Total 159 (delta 94), reused 91 (delta 91), pack-reused 51 Receiving objects: 100% (159/159), 55.57 KiB | 0 bytes/s, done. Resolving deltas: 100% (113/113), completed with 54 local objects. From https://github.com/zulip/zulip * branch refs/pull/1900/head -> FETCH_HEAD + git reset --hard FETCH_HEAD HEAD is now at 2bcd1d8 troubleshooting tip about provisioning
24.12.3 Fetch a pull request and rebase tools/fetch-rebase-pull-request is a short-cut for checking out a pull request locally in its own branch and then updating it with any changes from upstream/master with git rebase. First, make sure you are working in branch master. Then run the script with the ID number of the pull request as the first argument.
$ tools/fetch-rebase-pull-request 1913 + request_id=1913 + git fetch upstream pull/1913/head remote: Counting objects: 4, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (4/4), done. From https://github.com/zulip/zulip * branch refs/pull/1913/head -> FETCH_HEAD + git checkout upstream/master -b review-1913 Branch review-1913 set up to track remote branch master from upstream. Switched to a new branch 'review-1913' + git reset --hard FETCH_HEAD HEAD is now at 99aa2bf Add provision.py fails issue in common erros + git pull --rebase Current branch review-1913 is up to date.
24.12. Zulip-specific tools 149 Zulip Documentation, Release 1.4.0
150 Chapter 24. Git & GitHub Guide CHAPTER 25
Version control
25.1 Commit Discipline
We follow the Git project’s own commit discipline practice of “Each commit is a minimal coherent idea”. This discipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commit history a much more useful resource for developers trying to understand why the code works the way it does, which also helps a lot in preventing bugs. Coherency requirements for any commit: • It should pass tests (so test updates needed by a change should be in the same commit as the original change, not a separate “fix the tests that were broken by the last commit” commit). • It should be safe to deploy individually, or comment in detail in the commit message as to why it isn’t (maybe with a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checks in a future commit should be avoided – the security checks should be there from the beginning. • Error handling should generally be included along with the code that might trigger the error. • TODO comments should be in the commit that introduces the issue or functionality with further work required. When you should be minimal: • Significant refactorings should be done in a separate commit from functional changes. • Moving code from one file to another should be done in a separate commits from functional changes or even refactoring within a file. • 2 different refactorings should be done in different commits. • 2 different features should be done in different commits. • If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did, you probably should have just done 2 commits. When not to be overly minimal: • For completely new features, you don’t necessarily need to split out new commits for each little subfeature of the new feature. E.g. if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty of
151 Zulip Documentation, Release 1.4.0
options/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob of new code isn’t fun, so please be thoughtful about submitting things in reviewable units. • Don’t bother to split back end commits from front end commits, even though the backend can often be coherent on its own. Other considerations: • Overly fine commits are easily squashed, but not vice versa, so err toward small commits, and the code reviewer can advise on squashing. • If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, not writing a new “fix tests” commit on top of it. Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them; it’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix your history using git rebase -i. It can take some practice to get used to writing your commits with a clean history so that you don’t spend much time doing interactive rebases. For example, often you’ll start adding a feature, and discover you need to a refactoring part- way through writing the feature. When that happens, we recommend stashing your partial feature, do the refactoring, commit it, and then finish implementing your feature.
25.2 Commit Messages
First, check out these examples of commits with good commit messages. There are a few mistakes we see often that make commit messages less useful than they could be. • The first line of commit messages should be written in the imperative and be kept relatively short while concisely explaining what the commit does. And it should be clear about what part of the code is affected. For example: Good: gather_subscriptions: Fix exception handling bad input. zjsunit: Fix running stream_data and node tests individually. Add GitLab integration.
Bad: bugfix gather_subscriptions was broken fixing exception when given bad input fix bug #234.
• Use present-tense action verbs in your commit messages. Bad: gather_subscriptions: Fixing exception when given bad input. gather_subscriptions: Fixed exception when given bad input.
• Please use a complete sentence in the summary, ending with a period. • The rest of the commit message should be written in full prose and explain why and how the change was made. If the commit makes performance improvements, you should generally include some rough benchmarks showing that it actually improves the performance.
152 Chapter 25. Version control Zulip Documentation, Release 1.4.0
• When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue is automatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph of the commit message read e.g. “Fixes: #123.” • Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so that your commit message will be reasonably readable in git log in a normal terminal. • In your commit message, you should describe any manual testing you did in addition to running the automated tests, and any aspects of the commit that you think are questionable and you’d like special attention applied to.
25.2. Commit Messages 153 Zulip Documentation, Release 1.4.0
154 Chapter 25. Version control CHAPTER 26
Code style and conventions
26.1 Be consistent!
Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other code has actively bad style, fix it (in a separate commit). When in doubt, send an email to [email protected] with your question.
26.2 Lint tools
You can run them all at once with
./tools/lint-all
You can set this up as a local Git commit hook with
``tools/setup-git-repo``
The Vagrant setup process runs this for you. lint-all runs many lint checks in parallel, including • JavaScript (ESLint) • Python (Pyflakes) • templates • Puppet configuration • custom checks (e.g. trailing whitespace and spaces-not-tabs)
155 Zulip Documentation, Release 1.4.0
26.3 Secrets
Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function in zproject/settings.py to read secrets from /etc/zulip/secrets.conf.
26.4 Dangerous constructs
26.4.1 Misuse of database queries
Look out for Django code like this:
[Foo.objects.get(id=bar.x.id) for bar in Bar.objects.filter(...) if bar.baz<7]
This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of a list comprehension, write a single query using Django’s QuerySet API. If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t defer this optimization when performing schema changes, or else you may later find that it’s impossible.
26.4.2 UserProfile.objects.get() / Client.objects.get / etc.
In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Instead always use get_user_profile_by_{email,id}. There are 3 reasons for this: 1. It’s guaranteed to correctly do a case-inexact lookup 2. It fetches the user object from remote cache, which is faster 3. It always fetches a UserProfile object which has been queried using .selected_related(), and thus will perform well when one later accesses related models like the Realm. Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remote cache.
26.4.3 Using Django model objects as keys in sets/dicts
Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing with objects obtained from different database queries: For example, UserProfile.objects.only("id").get(id=17) in set([UserProfile. objects.get(id=17)]) is False You should work with the IDs instead.
26.4.4 user_profile.save()
You should always pass the update_fields keyword argument to .save() when modifying an existing Django model object. By default, .save() will overwrite every value in the column, which results in lots of race conditions where unrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfile object before the first thread wrote out its change.
156 Chapter 26. Code style and conventions Zulip Documentation, Release 1.4.0
26.4.5 Using raw saves to update important model objects
In most cases, we already have a function in zephyr/lib/actions.py with a name like do_activate_user that will correctly handle lookups, caching, and notifying running browsers via the event system about your change. So please check whether such a function exists before writing new code to modify a model object, since your new code has a good chance of getting at least one of these things wrong.
26.4.6 x.attr('zid') vs. rows.id(x)
Our message row DOM elements have a custom attribute zid which contains the numerical message ID. Don’t access this directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrong result, occasionally, just enough to make a bug that’s impossible to track down. You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Even in cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts in JavaScript where a string is needed, you can pass a number without any explicit conversion.
26.4.7 JavaScript var
Always declare JavaScript variables using var: var x=...;
In a function, var is necessary or else x will be a global variable. For variables declared at global scope, this has no effect, but we do it for consistency. JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts the same as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variables at the top of a function.
26.4.8 JavaScript for (i in myArray)
Don’t use it: [1], [2], [3]
26.4.9 jQuery global state
Don’t mess with jQuery global state once the app has loaded. Code like this is very dangerous:
$.ajaxSetup({ async: false }); $.get(...); $.ajaxSetup({ async: true }); jQuery and the browser are free to run other code while the request is pending, which could perform other Ajax requests with the altered settings. Instead, switch to the more general $.ajax_ function, which can take options like async.
26.4.10 State and logs files
Do not write state and logs files inside the current working directory in the production environment. This will not do what you expect, because the current working directory for the app changes every time we do a deploy. Instead, hardcode a path in settings.py – see SERVER_LOG_PATH in settings.py for an example.
26.4. Dangerous constructs 157 Zulip Documentation, Release 1.4.0
26.5 JS array/object manipulation
For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We used to use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer more choices. A quick conversion table:
$.each → _.each (parameters to the callback reversed) $.inArray → _.indexOf (parameters reversed) $.grep → _.filter $.map → _.map $.extend → _.extend
There’s a subtle difference in the case of _.extend; it will replace attributes with undefined, whereas jQuery won’t:
$.extend({foo: 2}, {foo: undefined}); // yields {foo: 2}, BUT... _.extend({foo: 2}, {foo: undefined}); // yields {foo: undefined}!
Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. If you’re doing this, you probably want _.find, _.every, or _.any, rather than ‘each’. Some Underscore functions have multiple names. You should always use the canonical name (given in large print in the Underscore documentation), with the exception of _.any, which we prefer over the less clear ‘some’.
26.6 More arbitrary style things
26.6.1 General
Indentation is four space characters for Python, JS, CSS, and shell scripts. Indentation is two space characters for HTML templates. We never use tabs anywhere in source code we write, but we have some third-party files which contain tabs. Keep third-party static files under the directory zephyr/static/third/, with one subdirectory per third-party project. We don’t have an absolute hard limit on line length, but we should avoid extremely long lines. A general guideline is: refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to 120 or so. Whitespace guidelines: • Put one space (or more for alignment) around binary arithmetic and equality operators. • Put one space around each part of the ternary operator. • Put one space between keywords like if and while and their associated open paren. • Put one space between the closing paren for if and while-like constructs and the opening curly brace. Put the curly brace on the same line unless doing otherwise improves readability. • Put no space before or after the open paren for function calls and no space before the close paren for function calls. • For the comma operator and colon operator in languages where it is used for inline dictionaries, put no space before it and at least one space after. Only use more than one space for alignment.
158 Chapter 26. Code style and conventions Zulip Documentation, Release 1.4.0
26.6.2 JavaScript
Don’t use == and != because these operators perform type coercions, which can mask bugs. Always use === and !==. End every statement with a semicolon. if statements with no braces are allowed, if the body is simple and its extent is abundantly clear from context and formatting. Anonymous functions should have spaces before and after the argument list:
var x= function (foo, bar) {//...
When calling a function with an anonymous function as an argument, use this style:
$.get('foo', function (data) { var x = ...; // ... });
The inner function body is indented one level from the outer function call. The closing brace for the inner function and the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate for calls with multiple anonymous functions or other arguments following them. Use
$(function () { ...
rather than
$(document).ready(function () { ...
and combine adjacent on-ready functions, if they are logically related. The best way to build complicated DOM elements is a Mustache template like zephyr/static/templates/ message.handlebars. For simpler things you can use jQuery DOM building APIs like so:
var new_tr = $('
Passing a HTML string to jQuery is fine for simple hardcoded things:
foo.append('
foo
');but avoid programmatically building complicated strings. We used to favor attaching behaviors in templates like so:
but there are some reasons to prefer attaching events using jQuery code: • Potential huge performance gains by using delegated events where possible • When calling a function from an onclick attribute, this is not bound to the element like you might think • jQuery does event normalization Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.
26.6. More arbitrary style things 159 Zulip Documentation, Release 1.4.0
26.6.3 HTML / CSS
Don’t use the style= attribute. Instead, define logical classes and put your styles in external files such as zulip. css. Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’t have to care if the tag type changes in the future. Don’t use inline event handlers (onclick=, etc. attributes). Instead, attach a jQuery event handler ($('#foo'). on('click', function () {...})) when the DOM is ready (inside a $(function () {...}) block). Use this format when you have the same block applying to multiple CSS styles (separate lines for each selector): selector1, selector2 { };
26.6.4 Python
• Scripts should start with #!/usr/bin/env python and not #/usr/bin/python (the right Python may not be installed in /usr/bin) or #/usr/bin/env python2.7 (bad for Python 3 compatibility). Don’t put a shebang line on a Python file unless it’s meaningful to run it as a script. (Some libraries can also be run as scripts, e.g. to run a test suite.) • Scripts should be executed directly (./script.py), so that the interpreter is implicitly found from the shebang line, rather than explicitly overridden (python script.py). • The first import in a file should be from __future__ import absolute_import, per PEP 328 • Put all imports together at the top of the file, absent a compelling reason to do otherwise. • Unpacking sequences doesn’t require list brackets:
[x, y]= xs # unnecessary x, y= xs # better
• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple. • When selecting by id, don’t use foo.pk when you mean foo.id. E.g.
recipient= Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
should be written as
recipient= Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
in case we ever change the primary keys.
26.6.5 Tests
All significant new features should come with tests. See testing.
26.6.6 Third party code
When adding new third-party packages to our codebase, please include “[third]” at the beginning of the commit message. You don’t necessarily need to do this when patching third-party code that’s already in tree.
160 Chapter 26. Code style and conventions CHAPTER 27
Python static type checker (mypy)
mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip is using mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written inside comments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip: user_dict={} # type: Dict[str, UserProfile] def get_user_profile_by_email(email): # type: (str) -> UserProfile ... # Actual code of the function here
You can learn more about it at: • The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotations used by mypy correctly. • The Python 2 type annotation syntax spec in PEP 484 • Using mypy with Python 2 code The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the ‘static-analysis’ build.
27.1 type_debug.py zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters of the decorated function and the return type whenever that function is called. This can help find out what parameter types a function is supposed to accept, or if parameters with the wrong types are being passed to a function. Here is an example using the interactive console:
>>> from zerver.lib.type_debug import print_types >>> >>> @print_types ... def func(x, y): ... return x+y
161 Zulip Documentation, Release 1.4.0
... >>> func(1.0,2) func(float, int) -> float 3.0 >>> func('a','b') func(str, str) -> str 'ab' >>> func((1,2), (3,)) func((int, int), (int,)) -> (int, int, int) (1, 2, 3) >>> func([1,2,3], [4,5,6,7]) func([int, ...], [int, ...]) -> [int, ...] [1, 2, 3, 4, 5, 6, 7] print_all prints the type of the first item of lists. So [int, ...] represents a list whose first element’s type is int. Types of all items are not printed because a list can have many elements, which would make the output too large. Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1: 'a', 2: 'b', 3: 'c'} will be printed as {int: str, ...}.
27.2 Zulip goals
Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays that way. Our current coverage is shown in Coveralls.
27.3 Installing mypy
If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3 virtualenv at zulip-py3-venv (mypy only supports Python 3). If it isn’t installed (e.g. because you haven’t reprovisioned recently), you can run tools/install-mypy to install it.
27.4 Running mypy on Zulip’s code locally
To run mypy on Zulip’s python code, run the command: tools/run-mypy
It will output errors in the same style as a compiler would. For example, if your code has a type error like this: foo=1 foo='1' you’ll get an error like this: test.py: note: In function"test": test.py:200: error: Incompatible types in assignment (expression has type"str",
˓→variable has type"int")
If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbott on your pull request (or email [email protected]) to get help; we’d love to both build a great trou- bleshooting guide in this doc and also help contribute improvements to error messages upstream.
162 Chapter 27. Python static type checker (mypy) Zulip Documentation, Release 1.4.0
Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinned version of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.
27.5 Excluded files
Since several Python files in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list of files to be excluded from the check for CI is present in tools/run-mypy. To run mypy on all Python files, ignoring the exclude list, you can pass the --all option to tools/run-mypy. tools/run-mypy--all
If you type annotate some of those files so that they pass without errors, please remove them from the exclude list.
27.6 Mypy is there to find bugs in Zulip before they impact users
For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a wide range of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to get to the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include: • A bug in any new type annotations you added. • A bug in the existing type annotations. • A bug in Zulip! • Some Zulip code is correct but confusingly reuses variables with different types. • A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project). Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way that makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @shar- maeklavya2 or @timabbott on the relevant pull request or issue on GitHub. If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHub repository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.
27.7 Annotating strings
In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to support languages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters like mathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCII characters. That’s why unicode was introduced in Python 2. But there are problems with the unicode and str system. Implicit conversions between str and unicode use the ascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people who always write in English. To minimize such implicit conversions, we should have a strict separation between str and unicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately it doesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking) insist that parameters passed to them are str. To make our code work correctly in Python 2, we have to identify strings which contain data which could come from non-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These strings
27.5. Excluded files 163 Zulip Documentation, Release 1.4.0 should be unicode. We also have to identify strings which should be str like Exception names, attribute names, parameter names, etc. Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static type checking will tell us if we are incorrectly mixing the two. However, unicode is not defined in Python 3. We want our code to be Python 3 compatible in the future. This can be achieved using ‘typing.Text’, a Python 2 and 3 compatibility type. typing.Text is defined as str in Python 3 and as unicode in Python 2. We’ll be using Text (instead of unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from typing import Text and using Text for annotation instead of doing import typing and using typing.Text for annotation, because Text is used so extensively for type annotations that we don’t need to be that verbose. Sometimes you’ll find that you have to convert strings from one type to another. zerver/lib/str_utils.py has utility functions to help with that. It also has documentation (in docstrings) which explains the right way to use them.
164 Chapter 27. Python static type checker (mypy) CHAPTER 28
Reviewing Zulip server code
This document is a brief discussion of what we look for when reviewing contributions to Zulip. It’s meant partially for developers who want to get their code merged faster, and partially for developers who have made successful pull requests already and would like to start participating in code review.
28.1 Things to look for
• The Travis CI build. The tests need to pass. One can investigate any failures and figure out what to fix by clicking on a red X next to the commit hash or the Detail links on a pull request. (Example: in #1219, click the red X next to f1f474e to see the build jobs for that commit, at least one of which has failed. Click on the link for Travis continuous integrations details to see the tests Travis ran on that commit, at least one of which failed, and go to one of the failing tests to see the error.) Since Coveralls’s data on changes isn’t always accurate, one should look at the test coverage situation directly. • Technical design. There are a lot of considerations here: security, migration paths/backwards compatibility, cost of new dependencies, interactions with features, speed of performance, API changes. Security is especially important and worth thinking about carefully with any changes to security-sensitive code like views. • User interface and visual design. If frontend changes are involved, the reviewer will check out the code, play with the new UI, and verify it for both quality and consistency with the rest of the Zulip UI. We highly encourage posting screenshots to save reviewers time in getting a feel for what the feature looks like – you’ll get a quicker response that way. • Error handling. The code should always check for invalid user input. User-facing error messages should be clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct the problem). • Testing. The tests should validate that the feature works correctly, and specifically test for common error con- ditions, bad user input, and potential bugs that are likely for the type of change being made. Tests that exclude whole classes of potential bugs are preferred when possible (e.g., the common test suite test_bugdown.py between the frontend and backend Markdown processors or the GetEventsTest test for buggy race condition handling).
165 Zulip Documentation, Release 1.4.0
Backend: we are trying to maintain ~100% test coverage on the backend, so backend changes should have negative tests for the various error conditions. Frontend: If the feature involves frontend changes, there should be frontend tests. See the test writing documentation for more details. • mypy annotations. New functions should be annotated using mypy and existing annotations should be updated. Use of Any, ignore, and unparameterized containser should be limited to cases where a more precise type cannot be specified. • Translation. Make sure that the strings are marked for translation. • Clear function, argument, variable, and test names. Every new piece of Zulip code will be read many times by other developers, and future developers will grep for relevant terms when researching a problem, so it’s important that variable names communicate clearly the purpose of each piece of the codebase. • Duplicated code. Code duplication is a huge source of bugs in large projects and makes the codebase difficult to understand, so we avoid significant code duplication wherever possible. Sometimes avoiding code duplication involves some refactoring of existing code; if so, that should usually be done as its own series of commits (not squashed into other changes or left as a thing to do later). That series of commits can be in the same pull request as the feature that they support, and we recommend ordering the history of commits so that the refactoring comes before the feature. That way, it’s easy to merge the refactoring (and minimize risk of merge conflicts) if there are still user experience issues under discussion for the feature itself. • Completeness. For refactorings, verify that the changes are complete. Usually one can check that efficiently using git grep, and it’s worth it, as we very frequently find issues by doing so. • Documentation updates. If this changes how something works, does it update the documentation in a corre- sponding way? If it’s a new feature, is it documented, and documented in the right place? • Good comments. It’s often worth thinking about whether explanation in a commit message or pull request discussion should be included in a comment, /docs, or other documentation. But it’s better yet if verbose explanation isn’t needed. We prefer writing code that is readable without explanation over a heavily commented codebase using lots of clever tricks. • Coding style. See the Zulip code-style documentation for details. Our goal is to have as much of this as possible verified via the linters and tests, but there’s always going to be unusual forms of Python/JavaScript style that our tools don’t check for. • Clear commit messages. See the Zulip version control documentation for details on what we look for.
28.2 Tooling
To make it easier to review pull requests, use our git tool tools/fetch-rebase-pull-request to check out a pull request locally and rebase it against master. If a pull request just needs a little fixing to make it mergeable, feel free to do that in a new commit, then push your branch to GitHub and mention the branch in a comment on the pull request. That’ll save the maintainer time and get the PR merged quicker.
28.3 Additional Resources
We also strongly recommend reviewers to go through the following resources. • The Gentle Art of Patch Review article by Sarah Sharp • Zulip & Good Code Review article by Sumana Harihareswara • Zulip Code of Conduct
166 Chapter 28. Reviewing Zulip server code CHAPTER 29
Testing and writing tests
29.1 Overview
Zulip has a full test suite that includes many components. The most important components are documented in depth in their own sections: • Django: backend Python tests • Casper: end-to-end UI tests • Node: unit tests for JS front end code • Linters This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot database issues, how to manually test the front end, and how to plan for the future upgrade to Python3. We also document how to manually test the app.
29.2 Running tests
Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter the Vagrant environment before running the tests: vagrant ssh cd/srv/zulip
Then, to run the full Zulip test suite, do this:
./tools/test-all
This runs the linter (tools/lint-all) plus all of our test suites; they can all be run separately (just read tools/ test-all to see them). You can also run individual tests which can save you a lot of time debugging a test failure, e.g.:
167 Zulip Documentation, Release 1.4.0
./tools/lint-all # Runs all the linters in parallel ./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube ./tools/test-backend BugdownTest # Run `test-backend --help` for more options ./tools/test-js-with-casper 09-navigation.js ./tools/test-js-with-node utils.js
The above setup instructions include the first-time setup of test databases, but you may need to rebuild the test database occasionally if you’re working on new database migrations. To do this, run:
./tools/do-destroy-rebuild-test-database
29.2.1 Possible testing issues
• When running the test suite, if you get an error like this:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_
˓→array(unknown, text, tsquery) does not exist LINE2:...ECT message_id, flags, subject, rendered_content, ts_match_l... ^
. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and the do-destroy-rebuild*-database scripts. • When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.
29.3 Schema and initial data changes
If you change the database schema or change the initial test data, you have to regenerate the pristine test database by running tools/do-destroy-rebuild-test-database.
29.4 Wiping the test databases
You should first try running: tools/do-destroy-rebuild-test-database If that fails you should try to do:
sudo-u postgres psql > DROP DATABASE zulip_test; > DROP DATABASE zulip_test_template;
and then run tools/do-destroy-rebuild-test-database
29.4.1 Recreating the postgres cluster
warning
168 Chapter 29. Testing and writing tests Zulip Documentation, Release 1.4.0
This is irreversible, so do it with care, and never do this anywhere in production. If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repair it, you can recreate it. On Ubuntu: sudo pg_dropcluster--stop 9.1 main sudo pg_createcluster--locale=en_US.utf8--start 9.1 main
29.5 Local browser testing (local app + web browser)
This section is about troubleshooting your local development environment. There is a separate manual testing doc that enumerates things you can test as part of manual QA.
29.5.1 Clearing the development database
You can use:
./tools/do-destroy-rebuild-database to drop the database on your development environment and repopulate your it with the Shakespeare characters and some test messages between them. This is run automatically as part of the development environment setup process, but is occasionally useful when you want to return to a clean state for testing.
29.5.2 JavaScript manual testing debug.js has some tools for profiling JavaScript code, including: • ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console. • ‘IterationProfiler‘: Profile part of looping constructs (like a for loop or $.each). You mark sections of the iteration body and the IterationProfiler will sum the costs of those sections over all iterations. Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. They both have profilers, but Chrome’s is a sampling profiler while Firebug’s is an instrumenting profiler. Using them both can be helpful because they provide different information.
29.6 Python 3 Compatibility
Zulip is working on supporting Python 3, and all new code in Zulip should be Python 2+3 compatible. We have converted most of the codebase to be compatible with Python 3 using a suite of 2to3 conversion tools and some manual work. In order to avoid regressions in that compatibility as we continue to develop new features in Zulip, we have a special tool, tools/check-py3, which checks all code for Python 3 syntactic compatibility by running a subset of the automated migration tools and checking if they trigger any changes. tools/check-py3 is run automatically in Zulip’s Travis CI tests (in the ‘static-analysis’ build) to avoid any regressions, but is not included in test-all since it is quite slow. To run tools/check-py3, you need to install the modernize and future Python packages (which are included in requirements/py3k.txt, which itself is included in requirements/dev.txt, so you probably already have these packages installed). To run check-py3 on just the Python files in a particular directory, you can change the current working directory (e.g. cd zerver/) and run check-py3 from there.
29.5. Local browser testing (local app + web browser) 169 Zulip Documentation, Release 1.4.0
170 Chapter 29. Testing and writing tests CHAPTER 30
Linters
30.1 Overview
Zulip does extensive linting of much of its source code, including Python/JavaScript files, HTML templates (Django/handlebars), CSS files, JSON fixtures, Markdown documents, puppet manifests, and shell scripts. For some files we simply check for small things like trailing whitespace, but for other files, we are quite thorough about checking semantic correctness. Obviously, a large reason for linting code is to enforce the Zulip coding standards. But we also use the linters to prevent common coding errors. We borrow some open source tools for much of our linting, and the links below will direct you to the official docu- mentation for these projects. • eslint • mypy • puppet (puppet provides its own mechanism for validating manifests) • pyflakes Zulip also uses some home-grown code to perform tasks like validating indentation in template files, enforcing coding standards that are unique to Zulip, allowing certain errors from third party linters to pass through, and exempting legacy files from lint checks.
30.2 Running the linters
If you run ./tools/test-all, it will automatically run the linters (with one small exception: it does not run mypy against scripts). You can also run them individually:
171 Zulip Documentation, Release 1.4.0
./tools/lint-all ./tools/run-mypy ./tools/run-mypy--scripts-only
Finally, you can rely on our Travis CI setup to run linters for you, but it is good practice to run lint checks locally. Note: The linters only check files that git tracks. Remember to git add new files before running lint checks. Our linting tools generally support the ability to lint files individually–with some caveats–and those options will be described later in this document. We may eventually bundle run-mypy into lint-all, but mypy is pretty resource intensive compared to the rest of the linters, because it does static code analysis. So we keep mypy separate to allow folks to quickly run the other lint checks.
30.3 General considerations
Once you have read the Zulip coding guidelines, you can be pretty confident that 99% of the code that you write will pass through the linters fine, as long as you are thorough about keeping your code clean. And, of course, for minor oversights, lint-all is your friend, not your foe. Occasionally, our linters will complain about things that are more of an artifact of the linter limitations than any actual problem with your code. There is usually a mechanism where you can bypass the linter in extreme cases, but often it can be a simple matter of writing your code in a slightly different style to appease the linter. If you have problems getting something to lint, you can submit an unfinished PR and ask the reviewer to help you work through the lint problem, or you can find other people in the Zulip Community to help you. Also, bear in mind that 100% of the lint code is open source, so if you find limitations in either the Zulip home-grown stuff or our third party tools, feedback will be highly appreciated. Finally, one way to clean up your code is to thoroughly exercise it with tests. The Zulip test documentation describes our test system in detail.
30.4 Lint checks
Most of our lint checks get performed by ./tools/lint-all. These include the following checks: • Check Python code with pyflakes. • Check JavaScript code with eslint. • Check Python code for custom Zulip rules. • Check non-Python code for custom Zulip rules. • Check puppet manifests with the puppet validator. • Check HTML templates for matching tags and indentations. • Check CSS for parsability. • Check JavaScript code for addClass calls. The remaining lint checks occur in ./tools/run-mypy. It is probably somewhat of an understatement to call “mypy” a “linter,” as it performs static code analysis of Python type annotations throughout our Python codebase. Our documentation on using mypy covers mypy in more detail. The rest of this document pertains to the checks that occur in ./tools/lint-all.
172 Chapter 30. Linters Zulip Documentation, Release 1.4.0
30.5 lint-all
Zulip has a script called lint-all that lives in our “tools” directory. It is the workhorse of our linting system, although in some cases it dispatches the heavy lifting to other components such as pyflakes, eslint, and other home grown tools. You can find the source code here. In order for our entire lint suite to run in a timely fashion, the lint-all script performs several lint checks in parallel by forking out subprocesses. This mechanism is still evolving, but you can look at the method run_parallel to get the gist of how it works.
30.5.1 Special options
You can use the -h option for lint-all to see its usage. One particular flag to take note of is the --modified flag, which enables you to only run lint checks against files that are modified in your git repo. Most of the “sub- linters” respect this flag, but some will continue to process all the files. Generally, a good workflow is to run with --modified when you are iterating on the code, and then run without that option right before commiting new code. If you need to troubleshoot the linters, there is a --verbose option that can give you clues about which linters may be running slow, for example.
30.5.2 Lint checks
The next part of this document describes the lint checks that we apply to various file types.
Generic source code checks
We check almost our entire codebase for trailing whitespace. Also, we disallow tab (\t) characters in all but two files. We also have custom regex-based checks that apply to specific file types. For relatively minor files like Markdown files and JSON fixtures, this is the extent of our checking. Finally, we’re checking line length in Python code (and hope to extend this to other parts of the codebase soon). You can use #ignorelinelength for special cases where a very long line makes sense (e.g. a link in a comment to an extremely long URL).
Python code
The bulk of our Python linting gets outsourced to the “pyflakes” tool. We call “pyflakes” in a fairly vanilla fashion, and then we post-process its output to exclude certain types of errors that Zulip is comfortable ignoring. (One notable class of error that Zulip currently tolerates is unused imports–because of the way mypy type annotations work in Python 2, it would be inconvenient to enforce this too strictly.) Zulip also has custom regex-based rules that it applies to Python code. Look for python_rules in the source code for lint-all. Note that we provide a mechanism to exclude certain lines of codes from these checks. Often, it is simply the case that our regex approach is too crude to correctly exonerate certain valid constructs. In other cases, the code that we exempt may be deemed not worthwhile to fix.
30.5. lint-all 173 Zulip Documentation, Release 1.4.0
JavaScript code
We check our JavaScript code in a few different ways: • We run eslint. • We perform custom Zulip regex checks on the code. • We verify that all addClass calls, with a few exceptions, explicitly contain a CSS class. The last check happens via a call to ./tools/find-add-class. This particular check is a work in progress, as we are trying to evolve a more rigorous system for weeding out legacy CSS styles, and the ability to quickly introspect our JS code for addClass calls is part of our vision.
Puppet manifests
We use Puppet as our tool to manage configuration files, using puppet “manifests.” To lint puppet manifests, we use the “parser validate” option of puppet.
HTML Templates
Zulip uses two HTML templating systems: • Django templates • handlebars Zulip has an internal tool that validates both types of templates for correct indentation and matching tags. You can find the code here: • driver: check-templates • engine: lib/template_parser.py We exempt some legacy files from indentation checks, but we are hoping to clean those files up eventually.
CSS
Zulip does not currently lint its CSS for any kind of semantic correctness, but that is definitely a goal moving forward. We do ensure that our home-grown CSS parser can at least parse the CSS code. This is a slightly more strict check than checking that the CSS is compliant to the official spec, as our parser will choke on unusual constructs that we probably want to avoid in our code, anyway. (When the parser chokes, the lint check will fail.) You can find the code here: • driver: check-css • engine: lib/css_parser.py
Markdown, shell scripts, JSON fixtures
We mostly validate miscellaneous source files like .sh, .json, and .md files for whitespace issues.
30.6 Philosophy
If you want to help improve Zulip’s system for linting, here are some considerations.
174 Chapter 30. Linters Zulip Documentation, Release 1.4.0
30.6.1 Speed
We want our linters to be fast enough that most developers will feel comfortable running them in a pre-commit hook, so we run our linters in parallel and support incremental checks.
30.6.2 Accuracy
We try to catch as many common mistakes as possible, either via a linter or an automated test.
30.6.3 Completeness
Our goal is to have most common style issues by caught by the linters, so new contributors to the codebase can efficiently fix produce code with correct style without needing to go back-and-forth with a reviewer.
30.6. Philosophy 175 Zulip Documentation, Release 1.4.0
176 Chapter 30. Linters CHAPTER 31
JavaScript unit tests
As an alternative to the black-box whole-app testing, you can unit test individual JavaScript files. If you are writing JavaScript code that manipulates data (as opposed to coordinating UI changes), then you probably modify existing unit test modules to ensure the quality of your code and prevent regressions. The JS unit tests are written to work with node. You can find them in frontend_tests/node_tests. Here is an example test from frontend_tests/node_tests/stream_data.js:
(function test_get_by_id() { stream_data.clear_subscriptions(); var id= 42; var sub={ name:'Denmark', subscribed: true, color:'red', stream_id: id }; stream_data.add_sub('Denmark', sub); sub= stream_data.get_sub('Denmark'); assert.equal(sub.color,'red'); sub= stream_data.get_sub_by_id(id); assert.equal(sub.color,'red'); }());
The names of the node tests generally align with the names of the modules they test. If you modify a JS module in static/js you should see if there are corresponding test in frontend_tests/node_tests. If there are, you should strive to follow the patterns of the existing tests and add your own tests.
31.1 HTML output
The JavaScript unit tests can generate output to be viewed in the browser. The best examples of this are in frontend_tests/node_tests/templates.js.
177 Zulip Documentation, Release 1.4.0
The main use case for this mechanism is to be able to unit test templates and see how they are rendered without the complications of the surrounding app. (Obviously, you still need to test the app itself!) The HTML output can also help to debug the unit tests. Each test calls a method named write_handlebars_output after it renders a template with similar data. This API is still evolving, but you should be able to look at existing code for patterns. When you run tools/test-js-with-node, it will present you with a message like “To see more output, open var/test-js-with-node/index.html.” Basically, you just need to open the file in the browser. (If you are running a VM, this might require switching to another terminal window to launch the open command.)
31.2 Coverage reports
You can automatically generate coverage reports for the JavaScript unit tests like this:
tools/test-js-with-node cover
Then open coverage/lcov-report/js/index.html in your browser. Modules we don’t test at all aren’t listed in the report, so this tends to overstate how good our overall coverage is, but it’s accurate for individual files. You can also click a filename to see the specific statements and branches not tested. 100% branch coverage isn’t necessarily possible, but getting to at least 80% branch coverage is a good goal.
31.3 Handling dependencies in unit tests
The following scheme helps avoid tests leaking globals between each other. First, if you can avoid globals, do it, and the code that is directly under test can simply be handled like this:
var search= require('js/search_suggestion.js');
For deeper dependencies, you want to categorize each module as follows: • Exercise the module’s real code for deeper, more realistic testing? • Stub out the module’s interface for more control, speed, and isolation? • Do some combination of the above? For all the modules where you want to run actual code, add a statement like the following to the top of your test file:
add_dependencies({ _:'node_modules/underscore/underscore.js', util:'js/util.js', Dict:'js/dict.js', Handlebars:'handlebars', Filter:'js/filter.js', typeahead_helper:'js/typeahead_helper.js', stream_data:'js/stream_data.js', narrow:'js/narrow.js' });
For modules that you want to completely stub out, please use a pattern like this:
set_global('page_params',{ email:'[email protected]' });
178 Chapter 31. JavaScript unit tests Zulip Documentation, Release 1.4.0
// then maybe further down global.page_params.email='[email protected]';
Finally, there’s the hybrid situation, where you want to borrow some of a module’s real functionality but stub out other pieces. Obviously, this is a pretty strong smell that the other module might be lacking in cohesion, but that code might be outside your jurisdiction. The pattern here is this:
// Use real versions of parse/unparse var narrow= require('js/narrow.js'); set_global('narrow',{ parse: narrow.parse, unparse: narrow.unparse });
// But later, I want to stub the stream without having to call super-
˓→expensive // real code like narrow.activate(). global.narrow.stream= function () { return 'office'; };
31.4 Creating new test modules
The nodes tests rely on JS files that use the module pattern. For example, to test the foobar.js file, you would first add the following to the bottom of foobar.js:
if (typeof module !=='undefined'){ module.exports= foobar; }
This makes foobar.js follow the CommonJS module pattern, so it can be required in Node.js, which runs our tests. Now create frontend_tests/node_tests/foobar.js. At the top, require the Node.js assert module, and the module you’re testing, like so:
var assert = require('assert'); var foobar= require('js/foobar.js');
And of course, if the module you’re testing depends on other modules, or modifies global state, you may need to review the section on handling dependencies above. Define and call some tests using the assert module. Note that for “equal” asserts, the actual value comes first, the expected value second.
(function test_somefeature() { assert.strictEqual(foobar.somefeature('baz'),'quux'); assert.throws(foobar.somefeature('Invalid Input')); }());
The test runner (index.js) automatically runs all .js files in the frontend_tests/node directory.
31.4. Creating new test modules 179 Zulip Documentation, Release 1.4.0
180 Chapter 31. JavaScript unit tests CHAPTER 32
Backend Django tests
32.1 Overview
Zulip uses the Django framework for its Python back end. We use the testing framework from django.test to test our code. We have over a thousand automated tests that verify that our backend works as expected. All changes to the Zulip backend code should be supported by tests. We enforce our testing culture during code review, and we also use coverage tools to measure how well we test our code. We mostly use tests to prevent regressions in our code, but the tests can have ancillary benefits such as documenting interfaces and influencing the design of our software. If you have worked on other Django projects that use unit testing, you will probably find familiar patterns in Zulip’s code. This document describes how to write tests for the Zulip back end, with a particular emphasis on areas where we have either wrapped Django’s test framework or just done things that are kind of unique in Zulip.
32.2 Running tests
Our tests live in zerver/tests/. You can run them with ./tools/test-backend. It generally takes about a minute to run the entire test suite. When you are in iterative mode, you can run individual tests or individual modules, following the dotted.test.name convention below:
cd/srv/zulip ./tools/test-backend zerver.tests.tests.WorkerTest
There are many command line options for running Zulip tests, such as a --verbose option. The best way to learn the options is to use the online help:
./tools/test-backend-h
We also have ways to instrument our tests for finding code coverage, URL coverage, and slow tests. Use the -h option to discover these features. We also have a --profile option to facilitate profiling tests.
181 Zulip Documentation, Release 1.4.0
Another thing to note is that our tests generally “fail fast,” i.e. they stop at the first sign of trouble. This is generally a good thing for iterative development, but you can override this behavior with the --nonfatal-errors option.
32.3 How to write tests.
Before you write your first tests of Zulip, it is worthwhile to read the rest of this document, and you can also read some of the existing tests in zerver/tests to get a feel for the patterns we use. A good practice is to get a “failing test” before you start to implement your feature. First, it is a useful exercise to understand what needs to happen in your tests before you write the code, as it can help drive out simple design or help you make incremental progress on a large feature. Second, you want to avoid introducing tests that give false positives. Ensuring that a test fails before you implement the feature ensures that if somebody accidentally regresses the feature in the future, the test will catch the regression. Another important files to skim are zerver/lib/test_helpers.py, which contains test helpers. zerver/lib/test_classes.py, which contains our ZulipTestCase and WebhookTestCase classes.
32.3.1 Setting up data for tests
All tests start with the same fixture data. (The tests themselves update the database, but they do so inside a transaction that gets rolled back after each of the tests complete. For more details on how the fixture data gets set up, refer to tools/setup/generate-fixtures.) The fixture data includes a few users that are named after Shakesepeare characters, and they are part of the “zulip.com” realm. Generally, you will also do some explicit data setup of your own. Here are a couple useful methods in ZulipTestCase: • common_subscribe_to_streams • send_message • make_stream • subscribe_to_stream More typically, you will use methods directly from the backend code. (This ensures more end-to-end testing, and avoids false positives from tests that might not consider ancillary parts of data setup that could influence tests results.) Here are some example action methods that tests may use for data setup: • check_send_message • do_change_is_admin • do_create_user • do_make_stream_private
32.4 Zulip Testing Philosophy
If there is one word to describe Zulip’s philosophy for writing tests, it is probably “flexible.” (Hopefully “thorough” goes without saying.) When in doubt, unless speed concerns are prohibitive, you usually want your tests to be somewhat end-to-end, partic- ularly for testing endpoints. These are some of the testing strategies that you will see in the Zulip test suite...
182 Chapter 32. Backend Django tests Zulip Documentation, Release 1.4.0
32.4.1 Endpoint tests
We strive to test all of our URL endpoints. The vast majority of Zulip endpoints support a JSON interface. Regardless of the interface, an endpoint test generally follows this pattern: • Set up the data. • Login with self.login() or set up an API key. • Use a Zulip test helper to hit the endpoint. • Assert that the result was either a success or failure. • Check the data that comes back from the endpoint. Generally, if you are doing endpoint tests, you will want to create a test class that is a subclass of ZulipTestCase, which will provide you helper methods like the following: • api_auth • assert_json_error • assert_json_success • client_get • client_post • get_api_key • get_streams • login • send_message
32.4.2 Library tests
For certain Zulip library functions, especially the ones that are not intrinsically tied to Django, we use a classic unit testing approach of calling the function and inspecting the results. For these types of tests, you will often use methods like self.assertEqual(), self.assertTrue(), etc., which come with unittest via Django.
32.4.3 Fixture-driven tests
Particularly for testing Zulip’s integrations with third party systems, we strive to have a highly data-driven approach to testing. To give a specific example, when we test our GitHub integration, the test code reads a bunch of sample inputs from a JSON fixture file, feeds them to our GitHub integration code, and then verifies the output against expected values from the same JSON fixture file. Our fixtures live in zerver/fixtures.
32.4.4 Mocks and stubs
We use mocks and stubs for all the typical reasons: • to more precisely test the target code • to stub out calls to third-party services
32.4. Zulip Testing Philosophy 183 Zulip Documentation, Release 1.4.0
• to make it so that you can run your tests on the airplane without wifi For mocking we generally use the “mock” library and use mock.patch as a context manager or decorator. We also take advantage of some context managers from Django as well as our own custom helpers. Here is an example:
with self.settings(RATE_LIMITING=True): with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock: api_result= my_webhook(request)
self.assertTrue(rate_limit_mock.called)
Follow this link for more information on the “settings” context manager.
32.4.5 Template tests
In zerver/tests/test_templates.py we have a test that renders all of our back end templates with a “dummy” context, to make sure the templates don’t have obvious errors. (These tests won’t catch all types of errors; they are just a first line of defense.)
32.4.6 SQL performance tests
A common class of bug with Django systems is to handle bulk data in an inefficient way, where the back end populates objects for join tables with a series of individual queries that give O(N) latency. (The remedy is often just to call select_related(), but sometimes it requires a more subtle restructuring of the code.) We try to prevent these bugs in our tests by using a context manager called queries_captured() that captures the SQL queries used by the back end during a particular operation. We make assertions about those queries, often simply asserting that the number of queries is below some threshold.
32.4.7 Event-based tests
The Zulip back end has a mechanism where it will fetch initial data for a client from the database, and then it will subsequently apply some queued up events to that data to the data structure before notifying the client. The EventsRegisterTest.do_test() helper helps tests verify that the application of those events via ap- ply_events() produces the same data structure as performing an action that generates said event. This is a bit esoteric, but if you read the tests, you will see some of the patterns. You can also learn more about our event system in the new feature tutorial.
32.4.8 Negative tests
It is important to verify error handling paths for endpoints, particularly situations where we need to ensure that we don’t return results to clients with improper authentication or with limited authorization. A typical test will call the endpoint with either a non-logged in client, an invalid API key, or missing input fields. Then the test will call assert_json_error() to verify that the endpoint is properly failing.
32.5 Testing considerations
Here are some things to consider when writing new tests: • Duplication We try to avoid excessive duplication in tests. If you have several tests repeating the same type of test setup, consider making a setUp() method or a test helper.
184 Chapter 32. Backend Django tests Zulip Documentation, Release 1.4.0
• Network independence Our tests should still work if you don’t have an internet connection. For third party clients, you can simulate their behavior using fixture data. For third party servers, you can typically simulate their behavior using mocks. • Coverage We have 100% line coverage on several of our backend modules. You can use the --coverage option to generate coverage reports, and new code should have 100% coverage, which generally requires testing not only the “happy path” but also error handling code and edge cases.
32.5. Testing considerations 185 Zulip Documentation, Release 1.4.0
186 Chapter 32. Backend Django tests CHAPTER 33
Web frontend black-box casperjs tests
These live in frontend_tests/casper_tests/. This is a “black box” test; we load the frontend in a real (headless) browser, from a real (development) server, and simulate UI interactions like sending messages, narrowing, etc. Since this is interacting with a real dev server, it can catch backend bugs as well. You can run this with ./tools/test-js-with-casper or as ./tools/test-js-with-casper 06-settings.js to run a single test file from frontend_tests/casper_tests/.
33.1 Debugging Casper.JS
Casper.js (via PhantomJS) has support for remote debugging. However, it is not perfect. Here are some steps for using it and gotchas you might want to know. To turn on remote debugging, pass --remote-debug to the ./frontend_tests/run-casper script. This will run the tests with port 7777 open for remote debugging. You can now connect to localhost:7777 in a Webkit browser. Somewhat recent versions of Chrome or Safari might be required. • When connecting to the remote debugger, you will see a list of pages, probably 2. One page called about:blank is the headless page in which the CasperJS test itself is actually running in. This is where your test code is. • The other page, probably localhost:9981, is the Zulip page that the test is testing—that is, the page running our app that our test is exercising. Since the tests are now running, you can open the about:blank page, switch to the Scripts tab, and open the running 0x-foo.js test. If you set a breakpoint and it is hit, the inspector will pause and you can do your normal JS debugging. You can also put breakpoints in the Zulip webpage itself if you wish to inspect the state of the Zulip frontend. You can also check the screenshots of failed tests at /tmp/casper-failure*.png. If you need to use print debugging in casper, you can do using casper.log; see http://docs.casperjs.org/en/latest/ logging.html for details.
187 Zulip Documentation, Release 1.4.0
An additional debugging technique is to enable verbose mode in the Casper tests; you can do this by adding to the top of the relevant test file the following:
var casper= require('casper').create({ verbose: true, logLevel:"debug" });
This can sometimes give insight into exactly what’s happening.
33.2 Writing Casper tests
Probably the easiest way to learn how to write Casper tests is to study some of the existing test files. There are a few tips that can be useful for writing Casper tests in addition to the debugging notes below: • Run just the file containing your new tests as described above to have a fast debugging cycle. • With frontend tests in general, it’s very important to write your code to wait for the right events. Before essen- tially every action you take on the page, you’ll want to use waitForSelector, waitUntilVisible, or a similar function to make sure the page or elemant is ready before you interact with it. For instance, if you want to click a button that you can select via #btn-submit, and then check that it causes success-elt to appear, you’ll want to write something like:
casper.waitForSelector("#btn-submit", function () { casper.click('#btn-submit') casper.test.assertExists("#success-elt"); });
This will ensure that the element is present before the interaction is attempted. The various wait functions supported in Casper are documented in the Casper here: http://docs.casperjs.org/en/latest/modules/casper.html# waitforselector and the various assert statements available are documented here: http://docs.casperjs.org/en/ latest/modules/tester.html#the-tester-prototype • The ‘waitFor’ style functions (waitForSelector, etc.) cannot be chained together in certain conditions without creating race conditions where the test may fail nondeterministically. For example, don’t do this:
casper.waitForSelector('tag 1'); casper.waitForSelector('tag 2');
Instead, if you want to avoid race condition, wrap the second waitFor in a then function like this:
casper.waitForSelector('tag 1'); casper.then(function () { casper.waitForSelector('tag 2'); });
• The selectors appearing in Casper tests are CSS3 selectors, which is a slightly different syntax from jQuery selectors. You can often save time by testing and debugging your selectors on the relevant page of the Zulip development app in the Chrome JavaScript console by using e.g. $$("#settings-dropdown") (this syntax is the CSS selector equivalent to querySelectorAll(), only available in the browser’s JavaScript console). You can learn more about these selectors and other JavaScript console tools here. • The test suite uses a smaller set of default user accounts and other data initialized in the database than the development environment; to see what differs check out the section related to options["test_suite"] in zilencer/management/commands/populate_db.py.
188 Chapter 33. Web frontend black-box casperjs tests Zulip Documentation, Release 1.4.0
• Casper effectively runs your test file in two phases – first it runs the code in the test file, which for most test files will just collect a series of steps (each being a casper.then or casper.wait... call). Then, usually at the end of the test file, you’ll have a casper.run call which actually runs that series of steps. This means that if you write code in your test file outside a casper.then or casper.wait... method, it will actually run be- fore all the Casper test steps that are declared in the file, which can lead to confusing failures where the new code you write in between two casper.then blocks actually runs before either of them. See this for more details about how Casper works: http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work
33.2. Writing Casper tests 189 Zulip Documentation, Release 1.4.0
190 Chapter 33. Web frontend black-box casperjs tests CHAPTER 34
Manual testing
As a general rule, we like to have automated tests for everything that can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they’re working on. This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip.
34.1 Basic Stuff
When testing Zulip manually, here are things to focus on: • The best bugs to catch are security/permissions bugs. • Don’t rush manual testing. Look for small details like display glitches. • Always test with multiple users (you can use incognito windows to facilitate this). • Always keep the inspector console open and watch for warnings or errors. • Be methodical about collecting information on bugs. (You will eventually want to create tickets, but you may want to consolidate your own notes before filing tickets.) You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her. The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc.
34.1.1 Message view
We mostly test the message view as part of testing everything else, but there are few things to specially test here.
191 Zulip Documentation, Release 1.4.0
Try using all the navigation hotkeys: • Up/k • Down/j • PgUp/K • PgDn/J/Spacebar • End (or fn-right-arrow on OSX) • also try scrolling aggressively with the mouse Try narrowing from the message view: • Hotkeys – use Esc to go to home – use s to narrow to a stream (select message first and verify in sidebar) – use S to narrow to the topic (and verify in sidebar) – use v to navigate to private messages • Click on the recipient bar – narrow to a stream – narrow to a topic – narrow to PMs with one user – narrow to a group PM • Click on the Zulip logo – narrow to a topic – click on the Zulip logo (and verify you’re in the home view)
34.1.2 Message editing
With message editing we mainly want to exercise topic changes. Here are some tasks: • Do lots of editing – send a message to the topic “original” – edit the message content – send two messages to the “original” stream – start to edit a message but then cancel – change the topic for the first message to “change1” (just this message) – narrow back to “original” – send one more message to the stream – change the topic for the last two messages to “change2” – narrow back to “original” – send two more messages to the stream
192 Chapter 34. Manual testing Zulip Documentation, Release 1.4.0
– edit the 2nd message on topic and change all messages to “change3” • Test UI entry points – hit “i” then down arrow to edit with the popup – use the popup using the mouse – enter edit mode using the pencil icon
34.1.3 Narrowing
Zulip uses the term “narrowing” to refer to opening different views of your messages, whether by clicking on sidebar options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course, you also want to see messages show up in the message pane. And, finally, you should make sure that no messages outside the narrow show up in Cordelia’s view. Important: Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use different streams for your testing. When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia through various narrows. Here are the main tasks for Hamlet (and each message gets sent several times): • Send Cordelia/Othello a PM. • Send Cordelia a PM. • Send Othello a PM. • Post to Verona/foo. • Post to Verona/bar. • Post to Denmark/foo. • Post to Denmark/foo and mention Cordelia. For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have Hamlet send new messages after each narrow): • Go to Home view. • Go to Private Messages view. • Go to Private Messages w/Hamlet. • Go to Private Messages w/Hamlet and Othello. • Go to Verona view. • Go to Verona/bar view. • Go to Verona/foo view. • Go to Denmark view. • Go to Denmark/foo view. There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs.
34.1. Basic Stuff 193 Zulip Documentation, Release 1.4.0
34.1.4 Composing messages
We have pretty good automated tests for our markdown processor, so manual testing is targeted more to other interac- tions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed. • Hotkeys – use r to reply to a stream message – use r to reply to a PM – use R to reply to the author of a PM – use R to reply to the author of a PM stream – use c to compose a stream message – use C to compose a new PM • Buttons – Narrow to a stream and click on “New stream message” – Narrow “Private Messages” and click on “New stream message” – Narrow to a stream and click on “New private message” – Narrow “Private Messages” and click on “New private message” • Topics – Compose/send a message to a stream with no topic. – Compose/send a message to a stream with a new topic. – Compose/send a message to a stream with autocomplete. – Compose/send a message to a stream manually typing an existing topic. • Formatting stuff – Use the “A” icon to get markdown help. – Use the eyeball icon to show a preview and send from preview mode. – Toggle in and out of preview before sending a message. – Use @-mention to mention Hamlet (and send him a message). – Use #**devel** syntax and send to Hamlet, then follow the link. – Create a bulleted list. – Use the emoji icon to find an emoji in the picker. • Attachments – Send a message with an attachment using the paperclip icon. – Send a message with multiple attachments. – Copy an image from the clipboard. – Use drag/drop from the desktop to upload an image. • Drafts – Start composing a message then click outside the compose box.
194 Chapter 34. Manual testing Zulip Documentation, Release 1.4.0
– Use “restore drafts” to restore the draft. – Start composing then use “Esc” to abort the message. – Use “restore drafts” to restore the draft. – Start composing a stream message and then abort using the little “x” icon in the compose box. – Click on “New private message” and restore the draft. (You should now be sending to a stream.) • Click to send – Turn off enter-to-send.
* Send a two-paragraph message using tab and enter. * Send a two-paragraph message using control-enter or command-enter. – Turn on enter-to-send.
* Hit enter to send.
34.1.5 Popover menus
For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is “too sticky.” Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view. Here are the things to test: • Stream sidebar menus – Stream settings (just make sure it goes there) – Narrow (and then have Hamlet send a message) – Pin/unpin (do both) – Compose (send a message to the stream) – Mark as read (scroll back and then have Hamlet send you a message) – Mute/unmute (do both) – Unsubscribe (and then go to Stream settings in the gear menu to resubscribe) – Choose custom color (play around with this) • Topic sidebar menus – Narrow (and then have Hamlet send a message) – Mute/unmute (try both) – Mark as read (scroll back and then have Hamlet send you a message) • Left-message-pane menus (click on person’s name) – Verify email – Verify date message sent – Send a PM (make sure compose box is filled out ok) – Narrow to PMs with – Narrow to PMs sent by
34.1. Basic Stuff 195 Zulip Documentation, Release 1.4.0
• Right-pane-pane menus (click on chevron when hovering) – use “i” hotkey to open the menu – Edit a message you sent (using the down-arrow key to navigate the popup) – View Source for somebody else’s message (make sure it’s not editable) – Reply (send a message) – Collapse/uncollapse (try both) – Mute/unmute (try both, watch left sidebar) – Link to this conversation • Buddy list chevron menus – Narrow to PMs with – Narrow to message sent by – Compose a message to
34.1.6 Sidebar filtering
This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that stream. • Streams filtering – Use “w” hotkey to open the search. – Filter on “d”. – Pin/unpin Denmark. – Clear filter. – Use “A” and “D” hotkeys to cycle through the streams. – Filter again and then click somewhere else. • Buddy list filtering – Use “q” hotkey to open the search. – Filter for Hamlet, Prospero, Othello, etc. – Log on Hamlet and log off Hamlet while filtering for Hamlet. – Log on/log off Hamlet while filtering for Othello. – Log on/log off Hamlet while not filtering at all. – Filter again and then click somewhere else.
34.1.7 Stream permissions
This is an important category to test, because we obviously do not want to have bugs where people can read messages on streams they should not have access to. The general flow here is for Hamlet to create the streams and verify that Cordelia has the correct visibility to them. First, we start off with “positive” tests.
196 Chapter 34. Manual testing Zulip Documentation, Release 1.4.0
• Positive tests – Have Hamlet create a public stream w/Cordelia subscribed and have him post a message to the stream. – Have Hamlet create a public stream without Cordelia and then...
* Have Hamlet post to the stream. * Have Cordelia subscribe to the stream. * Verify Cordelia can see the previous message. * Have Cordelia post a message to the stream. – Have Hamlet create an invite-only stream with Cordelia invited and test a two-way conversation between the two users. For negative tests, we want to dig a little deeper to find back doors for Cordelia to access the stream. Here are some techniques to try: • Try to have her compose a message to the stream by circumventing autocomplete. • Try to have her narrow to the stream using stream:foo in search. • Go to stream settings and see if the stream shows up. For public streams, it’s ok for Cordelia to know the stream exists, and she can subsequently subscribe. For private streams, she should not even know they exist (until she’s invited, of course). • Negative tests – Have Hamlet create a public stream without inviting Cordelia.
* Verify Cordelia can see the stream in her settings. * Verify Cordelia can’t compose a message to the stream. * Verify that Cordelia sees nothing when Hamlet posts to the stream. – Have Hamlet create a public stream with Cordelia, but then have Iago revoke her subscription using the admin page.
* Verify that the stream appears in Cordelia’s left sidebar and then goes away. * Try to have Cordelia view the stream using a sneaky search along the lines of stream:foo. – Have Hamlet create a private stream without inviting Cordelia.
* Verify Cordelia can’t compose a message to the stream.
34.1.8 Search
The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar reflects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search. Here are searches you should be able to do with autocomplete: • stream:design • stream:Verona topic:Verona1 • stream:Verona keyword • sent by me • @-mentions
34.1. Basic Stuff 197 Zulip Documentation, Release 1.4.0
• starred messages • messages sent by Hamlet • PMs with Hamlet • PMs with Hamlet matching keyword “foo” There are some things you can try that don’t come up in autocomplete: • -stream:Verona (exclude Verona) • stream:Verona stream:devel (should return no results) Miscellaneous: • Use the “/” hotkey to start a search. • Use the “x” icon to clear a search. • Use the “Esc” hotkey to clear a search.
34.1.9 Stream settings
Test various UI entry points into stream settings: • Use small gear menu in left sidebar, then filter to “devel”. • Use popover menu in left sidebar next to “devel”. • Use gear menu above buddy list and filter to “devel”. • Use gear menu and click on “devel.” • Use gear menu and then click on chevron menu next to “devel.” (I’m not sure why we still have the chevron at this writing.) Create new public stream “public1” and add Hamlet: • Type “public1” in the text box and then click “Create new stream.” • Select “People must be invited” and then verify you can’t select “Announce stream”. • Select “Anyone can join” again to make it be public. • Check the checkbox for Hamlet. • Hit the “Create” button. Test subscribe/unsubscribe: • Log in as Hamlet and go to his stream settings. • As Cordelia, unsubscribe from “public1” using the checkmark in the streams settings page. • Verify that Hamlet sees that Cordelia has unsubscribed (and the subscriber count should decrement). • As Cordelia, resubscribe to “public1.” • Verify Hamlet sees that change. As Cordelia, exercise different options in Create Stream dialog by creating streams s1, s2, s3, etc.: • s1: anyone can join, announce it, and add Hamlet using filter feature • s2: people must be invited • s3: anyone can join, don’t announce
198 Chapter 34. Manual testing Zulip Documentation, Release 1.4.0
• s4: check all, then uncheck all, then invite only Hamlet • s5: invite everybody but Hamlet • s6: – create the stream as public, but don’t subscribe anybody initially – then click on stream options to add Hamlet using “Add” button Test per-stream options: • Use “devel” stream and send a message to it • Do mute and unmute, have Hamlet send messages • Test notifications on/off, have Hamlet send messages • Test pin and unpin, view left sidebar • Change stream color, and then view the left sidebar and the Home message view • Verify stream subscriber counts in the main stream view
34.1.10 User Settings
You can modify per-user settings by choosing “Settings” in the gear menu. Do these tasks as Cordelia. • Your account – Change full name (Hamlet should see the name change) – Customize avatar – Deactivate account (and then log in as Iago to re-activate Cordelia) • Display settings – Right now, these unfortunately require reloads to take effect. – Default language (change to Spanish) – Display user list in narrow windows (verify by making window thinner) – Display time in 24-hour format (and then test going back to AM/PM) • Notifications – Stream Message
* turn off notifications at user level · create a new stream · have Hamlet send a message
* turn on notifications at user level · create a new stream · have Hamlet send a message · then turn off notifications for that stream · have Hamlet send another message – Private Messages and @-mentions
* Test Desktop/Audible options
34.1. Basic Stuff 199 Zulip Documentation, Release 1.4.0
* You can ignore other stuff for now • Bots/API key – Create a bot with a generic avatar and send it a PM – Create a bot with a custom avatar and send it a PM – Change your API key • Alert words – Create an alert word – Have Hamlet send you a message that includes the alert word • Zulip labs – Turn on auto-scroll to new messages (and have Hamlet send you one) – Turn on/off “Enable desktop notifications for new streams” and test. (We may eliminate this option soon.)
34.1.11 Keyboard Shorcuts
We mostly test keyboard shortcuts as part of other tasks. Here are the tasks for this section: • Use the ”?” hotkey to open the keyboard help • Proofread the dialog for typos. • Close the dialog. • Re-open the keyboard help using the gear menu. • Find a hotkey that you don’t frequently use and experiment with its usage.
34.1.12 Miscellaneous menu options
Make sure that these options launch appropriate help screens: • Proofread and try a couple random options: – Message formatting – Search operators • Make sure help launches in a separate browser tab: – Desktop and mobile apps – Integrations – API documentation
34.1.13 Inviting users/tutorial
Here are the tasks: • Invite [email protected] using the link beneath the buddy list but then don’t take further action. • Fully invite [email protected] using the gear menu.
200 Chapter 34. Manual testing Zulip Documentation, Release 1.4.0
• Go to the development console to get the login link for [email protected]. • Go through the signup flow. • Follow the tutorial. • Use the gear menu to log out. • Log back in as Cordelia (admittedly, this step doesn’t really QA much of our production code, since the login flow is customized for the development environment).
34.1.14 To be continued...
This document does not cover admin options yet. The main thing to do when testing the admin system is to verify that changes are synced to other users.
34.1. Basic Stuff 201 Zulip Documentation, Release 1.4.0
202 Chapter 34. Manual testing CHAPTER 35
Settings system
The page documents the Zulip settings system, and hopefully should help you decide how to correctly implement new settings you’re adding to Zulip. We have two types of administrative settings in Zulip: server settings (which are set via configuration files are apply to the whole Zulip installation), and realm settings (which are usually set via the /#administration page in the Zulip web application) and apply to a single Zulip realm/organization (which for most Zulip servers is the only realm on the server). Philosophically, the goals of the settings system are to make it convenient for: • Zulip server administrations to configure Zulip’s featureset for their server without needing to patch Zulip • Realm administrators to configure settings for their organization independently without needing to talk with the server administrator. • Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration.
35.1 Server settings
Zulip uses the Django settings system, which means that the settings files are Python programs that set a lot of variables with all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django code using e.g.: from django.conf import settings print(settings.EMAIL_GATEWAY_PATTERN)
Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), you can use e.g.:
$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN %s@localhost:9991
Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the ~1000 lines of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settings for server administrations. In a production environment, we have:
203 Zulip Documentation, Release 1.4.0
• /etc/zulip/settings.py (the template is in the Zulip repo at zproject/ prod_settings_template.py) is the main system administration facing settings file for Zulip. It contains all the server-specific settings, such as how to send outgoing email, the hostname of the Postgres database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in zproject/settings.py, and then override it here. • /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate_secrets.py as part of installation) contains secrets used by the Zulip installation. These are read using the standard Python ConfigParser, and accessed in zproject/settings.py by the get_secret function. All secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the get_secret function in zproject/settings.py. • zproject/settings.py is the main Django settings file for Zulip. It contains all the settings that are constant for all Zulip installations (e.g. configuration for logging, static assets, middleware, etc.), as well as default values for the settings the user would set in /etc/zulip/settings.py (you can look at the DEFAULT_SETTINGS dictionary to easily review the settings available). zproject/settings.py has a line from prod_settings import *, which has the effect of importing /etc/zulip/settings. py in a prod environment (via a symlink). In a development environment, we have zproject/settings.py, and additionally: • zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just im- ports prod_settings_template.py. • zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf. • zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend and Casper), which are applied on top of the development environment settings. When adding a new server setting to Zulip, you will typically add it in two or three places: • In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments. If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the user will add the value when they configure the feature). • In an appropriate section of zproject/prod_settings_template.py, with documentation in the com- ments explaining the settings’s purpose and effect. • Possibly also zproject/dev_settings.py, if the desired value of the setting for Zulip development environments is different from the default for production (and similarly for zproject/test_settings. py). Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’s features; they should be enabled by default in production if we expect most Zulip sites to want those settings.
35.1.1 Testing non-default settings
You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However, this only works for settings which are checked at runtime, not settings which are only accessed in initialization of Django (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.
35.2 Realm settings
Realm settings are preferred for any configuration that is a matter of organizational policy (as opposed to technical capabilities of the server). As a result, configuration options for user-facing functionality is almost always added as a
204 Chapter 35. Settings system Zulip Documentation, Release 1.4.0 new realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be replaced with realm settings: • NAME_CHANGES_DISABLED • INLINE_IMAGE_PREVIEW • ENABLE_GRAVATAR • Which authentication methods are allowed should probably appear in both places; in server settings indicating the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants to allow users to login with.
35.2. Realm settings 205 Zulip Documentation, Release 1.4.0
206 Chapter 35. Settings system CHAPTER 36
Queue processors
Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: • Asynchronously doing expensive operations like sending email notifications which can take seconds per email and thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm). • Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g. UserActivityInternal) which don’t have any immediate runtime effect. • Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. • Processing mobile push notifications and email mirroring system messages. • Processing various errors, frontend tracebacks, and slow database queries in a batched fashion. • Doing markdown rendering for messages delivered to the Tornado via websockets. Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s in critical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients. We use the pika library to interface with RabbitMQ, using a simple custom integration defined in zerver/lib/ queue.py.
36.1 Adding a new queue processor
To add a new queue processor: • Define the processor in zerver/worker/queue_processors.py using the @assign_queue decora- tor; it’s pretty easy to get the template for an existing similar queue processor. This suffices to test your queue worker in the Zulip development environment (tools/run-dev.py will automatically restart the queue pro- cessors and start running your new queue processor code). You can also run a single queue processor manually using e.g. ./manage.py process_queue --queue=user_activity.
207 Zulip Documentation, Release 1.4.0
• So that supervisord will known to run the queue processor in production, you will need to define a program entry for it in servers/puppet/modules/zulip/files/supervisor/conf.d/zulip.conf and add it to the zulip-workers group further down in the file. • For monitoring, you need to add a check that your worker is running to puppet/zulip/files/cron.d/rabbitmq- numconsumers if it’s a one-at-a-time consumer like user_activity_internal or a custom nagios check if it is a bulk processor like slow_queries.
36.2 Publishing events into a queue
You can publish events to a RabbitMQ queue using the queue_json_publish function defined in zerver/ lib/queue.py.
36.3 Clearing a RabbitMQ queue
If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue
./manage.py purge_queue user_activity
You can also use the amqp tools directly. Install amqp-tools from apt and then run: amqp-delete-queue--username=zulip--password='...'--server=localhost \ --queue=user_presence with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.
208 Chapter 36. Queue processors CHAPTER 37
Writing bots
This feature is still experimental. The contrib_bots system is a new part of Zulip that allows bot developers to write a large class of bots by simply reacting to messages. With bots, you can • intercept and view messages sent by users on Zulip • send out new messages With bots, you cannot • modify an intercepted message (you have to send a new message) • send messages on behalf of other users • intercept private messages (except for PMs that are sent to the bot) On this page you’ll find: • A step-by-step tutorial on how to deploy a bot. • A step-by-step tutorial on how to develop a bot. •A documentation of the bot API. • Common problems when developing/deploying bots and their solutions. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help with writing bots, please visit the Zulip chat, open an issue, or submit a pull request to share your ideas!
37.1 How to deploy a bot
This guide will show you how to deploy a bot on your running Zulip server. It presumes that you already have a fully implemented
209 Zulip Documentation, Release 1.4.0
1. Copy your bot
INFO:root:starting message handling... INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
˓→(1): localhost
• Congrats! Now, your bot should be ready to test on the streams you’ve subscribed it to.
37.1.1 Test the followup.py bot
1. Do the previous steps for the followup.py bot. 2. Create the followup stream. 3. Subscribe the bot to the newly created followup stream and a stream you want to use it from, e.g. social. 4. Send a message to the stream you’ve subscribed the bot to (other than followup). If everything works, a copy of the message should now pop up in the followup stream.
37.2 How to develop a bot
The tutorial below explains the structure of a bot
210 Chapter 37. Writing bots Zulip Documentation, Release 1.4.0
Every bot is built upon this structure:
class MyBotHandler(object): ''' A docstring documenting this bot. '''
def usage(self): return '''Your description of the bot'''
def triage_message(self, message, client): #add your code here
def handle_message(self, message, client, state_handler): # add your code here
handler_class= MyBotHandler
• The class name (in this case MyBotHandler) can be defined by you and should match the name of your bot. To register your bot’s class, adjust the last line handler_class = MyBotHandler to match your class name. • Every bot needs to implement the functions – usage(self) – triage_message(self, message, client) – handle_message(self, message, client) • These functions are documented in the next section.
37.3 Bot API
This section documents the functions every bot needs to implement and the structure of the bot’s config file.
37.3.1 usage usage(self) is called to retrieve information about the bot.
Arguments
• self - the instance the method is called on.
Return values
• A string describing the bot’s functionality
Example implementation
37.3. Bot API 211 Zulip Documentation, Release 1.4.0
def usage(self): return ''' This plugin will allow users to flag messages as being follow-up items. Users should preface messages with"@followup". Before running this, make sure to create a stream called"followup" that your API user can send to. '''
37.3.2 triage_message triage_message(self, message, client) is called when a message was sent.
Arguments
• self - the instance the method is called on • message - a dictionary containing information about the message, e.g. – content - the content of the message – content_type - the type of the content, e.g. ‘text/x-markdown’ for normal messages – display_recipient - the name of the stream the message is sent to (string) – is_mentioned - is the bot pinged with an ‘@’ in the message? (boolean) – sender_email - email of the sender (string) – sender_full_name - full name of the sender (string) – subject - topic of the message (string) – timestamp - when was the message sent (integer) • client - contains information about this bot – client.full_name - name of the bot account – client.email - email of the bot account
Return values
• True if the bot should react to this message • False otherwise
Example implementation def triage_message(self, message, client): original_content= message['content'] if message['display_recipient'] =='followup': return False is_follow_up= (original_content.startswith('@followup') or original_content.startswith('@follow-up')) return is_follow_up
212 Chapter 37. Writing bots Zulip Documentation, Release 1.4.0
37.3.3 handle_message handle_message(self, message, client) is called when triage_message returns true, handles user message.
Arguments
• self - the instance the method is called on. • message - a dictionary describing a Zulip message • client - used to interact with the server, e.g. to send a message – use client.send_message(message) to send a message • state_handler - used to save states/information of the bot beta – use state_handler.set_state(state) to set a state (any object) – use state_handler.get_state() to retrieve the state set; returns a NoneType object if no state is set
Return values
None.
Example implementation
def handle_message(self, message, client, state_handler): original_content= message['content'] original_sender= message['sender_email'] new_content= original_content.replace('@followup', 'from %s:'% (original_sender,))
client.send_message(dict( type='stream', to='followup', subject=message['sender_email'], content=new_content, ))
37.3.4 Configuration file
[api] key=
• key - the API key you created for the bot; this is how Zulip knows the request is from an authorized user. • email - the email address of the bot, e.g. [email protected] • site - your development environment URL; if you are working on a development environment hosted on your computer, use localhost:9991
37.3. Bot API 213 Zulip Documentation, Release 1.4.0
37.4 Common problems
• I modified my bot’s code, yet the changes don’t seem to have an effect. – Ensure that you restarted the run.py script. • My bot won’t start – Ensure that your API config file is correct (download the config file from the server). – Ensure that you bot script is located in zulip/contrib_bots/bots/
214 Chapter 37. Writing bots CHAPTER 38
Custom Apps
38.1 Definition
Zulip defines a “custom app” to be a piece of code that runs in the Zulip ecosystem, but which is not part of the core Zulip codebase. Custom apps are mostly synonymous with “bots” and “integrations” in the Zulip ecosystem. We currently do not support any kind of browser plugin model.
38.2 Problem statement
Zulip wants to enable people in the world to author custom apps with the following goals in mind: • Simple custom apps should be simple to write and deploy. • Custom app authors should be able to easily distribute their work. • Zulip should provide deployment support for mature, general-purpose bots, ideally either within organizations (a Zulip admin can vet her own custom apps and easily deploy them across upgrade cycles) or across organizations (custom apps get distributed with the Zulip tarball). This document describes Zulip’s current infrastructure, as well as laying out a roadmap for some future features.
38.3 A quick note on bots/integrations
As noted earlier, a custom app is just a generic term for what we often call bots or integrations. We recognize that bots and integrations can have different connotations. A bot typically spends most of its time responding to Zulip messages. An integration usually represents an app that interacts with some large third party system like an issue tracker. We will use both terms in this document in an informal sense, but from an architecture standpoint, we treat bots and integrations as essentially two shades of the same color. Many integrations are implemented as “bots.” Likewise, any bot that does stuff outside of Zulip acts as an “integration.”
215 Zulip Documentation, Release 1.4.0
Since the line between what a “bot” is and what an “integration” is can get very blurry, we try to be informal about “bots/integrations” and more formal about how “custom apps” actually function within the system.
38.4 Categories of custom apps
38.4.1 Stimulus/response and read/write
At the end of the day, most useful apps respond to some stimulus and produce a response. In the Zulip universe, Zulip can be the source of the stimulus, or the target of the response, or both. Along those lines, we divide custom apps into these three types: •A Zulip Reader uses activity on Zulip to stimulate an external response. An example here would be a follow-up bot that sees messages with the alert word “@todo” on a stream and then adds a task to a third party todo-list tool. •A Zulip Writer reacts to external stimuli and generates Zulip responses. An example here might be a build bot that gets triggered by an automated code build finishing and then writes “build finished” to a Zulip stream. •A Zulip Read/Writer reacts to a stimulus from Zulip by responding to Zulip. An example here would be a math bot that sees a message saying “compute 2+2” and responds with “2+2=4” on the same stream or back to the user in a PM. The above three classifications represent kind of a Zulip-centric view of the universe, but we should put ourselves in the shoes of somebody “out in the world.” •A World Reader is an app that gets some stimulus from the outside world and produces a response in Zulip. (So, a world reader is a Zulip writer.) •A World Writer is an app that gets some stimulus from Zulip and produces a response in the outside world. (So, a world writer is a Zulip reader.) Some things are a little outside of the scope of this document. We could plausibly extend Zulip some day to host World Reader/Writer apps that don’t even write Zulip messages but simply use Zulip as a kind of middleware platform. More in the short term, we will have custom apps that may read/write from multiple sources. For example, a meeting bot may take input from both a cron job and a Zulip stream, and it may write to both a Zulip stream and a third party calendar tool. For the scope of this document, we won’t spend a lot of time talking about how to build these types of apps, but we are aware that any solution needs to accommodate multiple sources and targets.
38.4.2 World Reader/Zulip Reader
Finally, we set the stage for how we talk about custom apps in terms of these two broad categories: •A World Reader responds to stimuli from the outside world (and typically produces a response in Zulip). •A Zulip Reader responds to stimuli from Zulip conversations (and typically produces a response in the outside world). Again, we recognize that there can be overlap between those two categories for complex custom apps, but we mostly leave it as an exercise for the reader how to implement those apps.
38.4.3 Other classifications
We discussed one dimension for classifying custom apps, which is whether they are world-readers or Zulip-readers. Here we cover a few other classification schemes briefly:
216 Chapter 38. Custom Apps Zulip Documentation, Release 1.4.0
• Generality Does the custom app have a specific use case or a general one? The spectrum here could run from a bot that Alice runs to update a text file on her laptop (specific) to a Twitter Bot that is optionally deployed on all Zulip realms (general). • Authorship Who wrote the custom app? Was it written by contributors to the Zulip project? • Maturity How well tested is the custom app? Is it just a prototype? Has it been sanctioned by an open source community? Has it been vetted by Zulip developers? • Deployment Where does the custom app run? Does it run on Alice’s laptop? Does it run on a Zulip server? Does it run as a plugin on third party infrastructure? • Authorization What streams are the custom app allowed to read and write from? Which users can the custom app interact with? • Identity How does the custom app identify itself on Zulip? How does it identify itself to the outside world? • Third party We call the non-Zulip target or source of a custom app the “world.” The “world” could be almost anything, ranging from an electronic device or text file to a large third-party system like Twitter or GitHub. A lot of the classification schemes are interrelated. Here are some examples: • For specific-purpose custom apps, authors may be happy to just deploy them on their own hardware. For general- use custom apps, authors may want to have them deployed on the Zulip server with super-user capabilities. • As a custom app becomes more well-tested and well-vetted, the author will likely upgrade its deployment over time. At first the author may run the custom app on their laptop, then they may find dedicated hardware, and then finally they contribute the app to the Zulip project so that Zulip admins can deploy the app on Zulip servers. • The nature of the third party will influence the deployment strategy. If I have a little home-grown gadget that can turn off the lights in my kitchen, I may run a custom app on my laptop that reads my PMs for “turn-off-the-light” messages. If I write a generic custom app that needs to update a third party corporate system based on Zulip events, I may want to deploy code to a public webserver or try to get my code to be part of the Zulip project itself.
38.5 World Reader
A World Reader custom app is an app that responds to stimuli from the world outside of Zulip. It typically functions as a Zulip Writer and posts some kind of message to a Zulip stream or user to alert people of world events. Here are some example stimuli: • A Travis build finishes. • Somebody tweets on Twitter. • A hardware sensor notices a temperature increase. • A pull request is submitted to GitHub. • A cron job gets started on your laptop to send a reminder. • Nagios detects a system anomaly. Setting aside issues of how a custom app is constructed or deployed, you basically have to solve these problems: • Detect events. • Translate events into Zulip messages. • Post the messages to Zulip.
38.5. World Reader 217 Zulip Documentation, Release 1.4.0
38.5.1 Zulip integrations
Zulip actually supports a bunch of integrations out-of-the-box that perform as World Readers. The three different integration models basically differ in where they perform the main functions of a World Reader.
Webhook integrations
In a webhook integration, the deployment model is usually this:: 3rd party hardware: • detect event • send data to Zulip webhook Zulip: • support webhook endpoint • translate event to messages • internally post messages One current limitation of our system is that we don’t have a great way to deploy prototypes of webhook-based custom apps before Zulip has vetted the translation and added an official endpoint. Maybe we could set up some kind of webserver that can run translation code outside of Zulip and externally post the messages, and we could think about how to structure the code so that it is easy to eventually turn it into a Zulip-hosted integration.
Python scripts
In script integrations, the deployment model is usually this: Custom app author’s hardware: • detect event by polling a third party system • translate event in the script • externally post messages These type of integrations are typically easy to prototype, but they can be harder to deploy in production settings, since we rely on the authors to run their own scripts. In some cases authors might want to at least move the translation/posting code to live on Zulip, by contributing that code to Zulip as a server-side integration. Then, there would still be the challenge of detecting events in the third party system, where maybe the user submits a patch to the third party as well.
Plugin integrations
In plugin integrations, the deployment model is usually this: Third party system (driver): • detect event Third party system (plugin): • further detect/triage event • translate event
218 Chapter 38. Custom Apps Zulip Documentation, Release 1.4.0
• externally post to Zulip For third parties that have a plugin model, there are often other issues at play, like the plugins may need to be written in a non-Python language like Ruby. There are probably still some scenarios, however, where a lot of the logic for translation could be moved to a Zulip-side integration, and then we supply very thin client code for the plugin.
38.6 Zulip Reader
A Zulip Reader custom app gets stimuli from Zerver itself. Most Zulip Reader apps are packaged/advertised more as what people commonly call “bots” than as “integrations.” (But sometimes what is currently a “bot” should really be deployed more like an “integration” in an ideal Zulip universe.) Example custom Zulip Reader apps can be serious or whimsical. Serious • A user tags a message with an alert word like @followup or @ticket. • A user needs help computing something, like a simple math expression or a timezone conversion. •A World Reader custom app posts something to a Zulip stream that we want to cross-post to another external system. • A user wants the custom app to query the outside world, like look up the weather or search Wikipedia. • A bot collects RSVPs for an event. • A bot conducts a user survey. Whimsical • A user wants to see a random quote of the day or a random cat fact. • A user wants to tell the office telepresence robot to “turn left.” Setting aside whether a custom app is performing a serious or whimsical function, there are a few different types of Zulip Readers: • Some readers will do simple local computations and post right back to Zulip. • Some readers will do more expensive/web-related computations like searching Wikipedia, but then post right back to Zulip. • Some readers will mutate the outside world in some way, like posting messages to third party APIs or controlling hardware. • Some readers will do some combination of the prior bullets.
38.7 Deployment issues
Zulip currently provides only minimal deployment support for Zulip Reader custom apps: • It ships with a few native server-side bots like the welcome bot and the notifications bot. (These are nice to have, but they are so tightly integrated into the Zulip core that they don’t act as great examples for future app authors, and they are not easy to extend/customize.) • Zulip does ship an API client that can conveniently read a .zuliprc file, poll for incoming messages/events, and post new messages to the Zulip server.
38.6. Zulip Reader 219 Zulip Documentation, Release 1.4.0
38.7.1 Local deployment
If you download the API client and write a bot that reads from Zulip, you face the following challenges if you deploy your code on your own devices: • It can be difficult to keep the app running 24/7. • You may have latency issues connecting to the server. • If you want super-user permissions, you have to secure the API key. • Without integration to the Zulip server, the app may spin needlessly during upgrades. • If you’ve written a personal-use bot, it can be difficult to distribute code to your friends and have them be able to deploy it. • If you’ve written a general-use bot, it may be difficult to persuade your admin to give you a superuser account. We want to make it easier to deploy Zulip Readers on Zulip hardware. The following document talks about how we want to enable this from a code structuring standpoint: Writing contrib bots This document, on the other hand, is more about designing the Zulip backend system to support eventual deployment of reader apps on the Zulip server. Before we talk about server-side apps, we should consider an intermediate solution.
38.7.2 Non-Zulip dedicated hardware
There are some scenarios, mostly with general-purpose “serious” custom apps, where an app author might use the following development process: • Create a prototype and deploy it locally. • Publicize the app and deploy it on non-Zulip hardware. • Contribute the app to the Zulip distribution, so that admins can run it Zulip-side. To give a concrete example, let’s say that I work for a company that is building an issue tracker, and we want to offer Zulip support. I would start by writing a Zulip Reader that scans for the alert word @ticket on certain public Zulip streams, and part of that app would have logic to post to my company’s issue-tracking API. Once I’m confident in my prototype, I will probably run it on dedicated company hardware that might already have tight physical security, 24/7 IT monitoring, etc. But what if I don’t have this kind of infrastructure available to me? Typically what I will do instead is rent time on some kind of hosting service. Some hosting platforms are basically just remote Unix systems, but others are more oriented toward hosting web apps. Zulip’s current roadmap assumes that authors will likely gravitate toward web-based solutions (even if it’s just running a web server on their own Unix host in the cloud). Zulip intends to offer support for “outgoing webhooks.” The term “outgoing webhook” can be confusing, depending on your perspective, but it simply means that an HTTP request is outgoing from Zulip, so that it will hit a web endpoint that runs a third-party custom app. Zulip will allow the custom app author, probably with the help of a Zulip admin, to configure Zulip to send a subset of Zulip messages to the author’s web endpoint, and then the protocol for the custom app will to read the HTTP request and send some kind of HTTP response that optionally results in a message being written to Zulip. Meanwhile, the custom app can mutate the “world” as it sees fit.
220 Chapter 38. Custom Apps Zulip Documentation, Release 1.4.0
38.7.3 Zulip-side support for reader apps
Even for app authors that have access to dedicated hardware, there would be several advantages to running Zulip Readers under the same umbrella as the core Zulip system. • Your app will automatically inherit the uptime of the Zulip server itself (in terms of hardware availability). • There will be no network latency between the app and the server. • Securing apps to have superuser permissions will be less problematic. • Keeping your app in sync with Zulip upgrades could become more automatic. • Allowing multiple users in your realm to run their own copies of personal-use bots would be easier to administer. The only problem with the above bullets is that we haven’t built out any of that infrastructure yet. We do have pending PR #1393, which addresses some of the issues that might come up. In order to run apps inside the Zulip server, we basically need to solve the problems below. (One assumption is that we don’t run apps truly in-process.) • Contributions: We need a process for users to contribute code. • Configuration/Discovery: We need Zulip to be able to find which apps are allowed to run for a particular deployment. (The admin may choose to run only a subset of contributed apps.) • Queuing: We need to queue up events for readers, with some possible optimizations to scan for alert words during the in-process part of the call. • Drivers: We need a generic driver that can pull events off of a queue and hand them off to our specific reader objects. • Nannying: We need to launch readers with some kind of supervisord-like nannying. • Pausing: We probably need a way to pause/stop readers without stopping the Zulip main processes. (At first this may just be part of solving the nanny problem.) • Identity: We need to identify reader instances as specific Zulip users (non-owned bot, human-owned bot, or human). • Superusers: We may need some readers to have users with special privileges like being auto-subscribed to all public streams. • Read-only: We may need some readers at the other end of the spectrum to be highly locked down, e.g. enforce that they truly only have read access to Zulip messages. • UI: We will want to provide some UI features that give admins and/or regular users visibility into which server- side apps are running.
38.7. Deployment issues 221 Zulip Documentation, Release 1.4.0
222 Chapter 38. Custom Apps CHAPTER 39
Unread counts and the pointer
When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you? Conceptually, Zulip takes you to the place where you left off (e.g. the first unread message), not the most recent messages, to facilitate reviewing all the discussions that happened while you were away from your computer. The scroll position is then set to keep that message in view and away from both the top and bottom of the visible section of messages. But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currently decides which message to select, along with some notes on improvements we’d like to make to the model. First a bit of terminology: • “Narrowing” is the process of filtering to a particular subset of the messages the user has access to. • The blue cursor box (the “pointer”) is around is called the “selected” message. Zulip ensures that the currently selected message is always in-view.
39.1 Pointer logic
39.1.1 Recipient bar: message you clicked
If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list at the top of a group of messages), Zulip will select the message you clicked on. This provides a nice user experience where you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly the same scroll position in the window after the narrowing as it was at before.
39.1.2 Search or sidebar click: unread/recent matching narrow
If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulip will instead select the first unread message matching that narrow, or if there are none, the most recent messages matching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enough messages you’ev seen before still in view at the top to provide you with context), which is usually what you want.
223 Zulip Documentation, Release 1.4.0
(When finding the “first unread message”, Zulip ignores unread messages in muted streams or in muted topics within non-muted streams.)
39.1.3 Unnarrow: previous sequence
When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selected in the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped forward to the first unread and non-muted message in the home view (or the bottom of the feed if there is none). This makes for a nice experience reading threads via the home view in sequence.
39.1.4 New home view: “high watermark”
When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /), Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic around unnarrowing in the last bullet, this is usually just before the first unread message in the home view, but if you never go to the home view, or you leave messages unread on some streams in your home view, this can lag. We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.
39.1.5 Narrow in a new tab: closest to pointer
When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer, which is what you would have got had you loaded the browser window to your home view and then clicked on the nearest message matching your narrow (which might have been offscreen). We plan to change this to match the Search/sidebar behavior.
39.1.6 Forced reload: state preservation
When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when a new version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preserve the state – what (if any) narrow the user was in, the selected message, and even exact scroll position! For more on the user experience philosophy guiding these decisions, see the architectural overview.
39.2 Unread count logic
How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might use the product. The algorithm is as follows: • Any message which is selected or above a message which is selected is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. • If the whitespace at the very bottom of the feed is in view, all messages in view are marked as read. These two simple rules, combined with the pointer logic above, end up matching user expectations well for whether the product should treat them as having read a set of messages (or not).
224 Chapter 39. Unread counts and the pointer CHAPTER 40
Markdown implementation
Zulip has a special flavor of Markdown, currently called ‘bugdown’ after Zulip’s original name of “humbug”. End users are using Bugdown within the client, not original Markdown. Zulip has two implementations of Bugdown. The backend implementation at zerver/lib/bugdown/ is based on Python-Markdown and is used to authoritatively render messages to HTML (and implements slow/expensive/complex features like querying the Twitter API to render tweets nicely). The frontend implementation is in JavaScript, based on marked.js( static/js/echo.js), and is used to preview and locally echo messages the moment the sender hits enter, without waiting for round trip from the server. The JavaScript markdown implementation has a function, echo.contains_bugdown, that is used to check whether a message contains any syntax that needs to be rendered to HTML on the backend. If echo. contains_bugdown returns true, the frontend simply won’t echo the message for the sender until it receives the rendered HTML from the backend. If there is a bug where echo.contains_bugdown returns false incorrectly, the frontend will discover this when the backend returns the newly sent message, and will update the HTML based on the authoritative backend rendering (which would cause a change in the rendering that is visible only to the sender shortly after a message is sent). As a result, we try to make sure that echo.contains_bugdown is always correct.
40.1 Testing
The Python-Markdown implementation is tested by zerver/tests/test_bugdown.py, and the marked.js im- plementation and echo.contains_bugdown are tested by frontend_tests/node_tests/echo.js.A shared set of fixed test data (“test fixtures”) is present in zerver/fixtures/bugdown-data.json, and is automatically used by both test suites; as a result, it the preferred place to add new tests for Zulip’s markdown system.
40.2 Changing Zulip’s markdown processor
When changing Zulip’s markdown syntax, you need to update several places: • The backend markdown processor (zerver/lib/bugdown/__init__.py).
225 Zulip Documentation, Release 1.4.0
• The frontend markdown processor (static/js/echo.js and sometimes static/third/marked/ lib/marked.js), or echo.contains_bugdown if your changes won’t be supported in the frontend processor. • If desired, the typeahead logic in static/js/composebox_typeahead.js. • The test suite, probably via adding entries to zerver/fixtures/bugdown-data.json. • The in-app markdown documentation (templates/zerver/markdown_help.html). • The list of changes to markdown at the end of this document. Important considerations for any changes are: • Security: A bug in the markdown processor can lead to XSS issues. For example, we should not insert unsani- tized HTML from a third-party web application into a Zulip message. • Uniqueness: We want to avoid users having a bad experience due to accidentally triggering markdown syntax or typeahead that isn’t related to what they are trying to express. • Performance: Zulip can render a lot of messages very quickly, and we’d like to keep it that way. New regular expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about expensive computations or third-party API requests. • Database: The backend markdown processor runs inside a Python thread (as part of how we implement timeouts for third-party API queries), and for that reason we currently should avoid making database queries inside the markdown processor. This is a technical implementation detail that could be changed with a few days of work, but is important detail to know about until we do that work. • Testing: Every new feature should have both positive and negative tests; they’re easy to write and give us the flexibility to refactor frequently.
40.3 Zulip’s Markdown philosophy
Note that this discussion is based on a comparison with the original Markdown, not newer Markdown variants like CommonMark. Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis to bug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that it helps more than getting in the way. The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to fix the formatting after typing it the first time. While that’s basically fine when writing a blog, it gets annoying very fast in a chat product; even though you can edit messages to fix formatting mistakes, you don’t want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip: • What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it to render in a reasonable way? This is the “accidental Markdown syntax” problem, common with Markdown syntax like the italics syntax interacting with talking about char *s. • What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figure substantially. Both of these are minor issues for most products using Markdown, but they are major problems in the instant messaging context, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’s Markdown strategy is based on the principles of giving users the power they need to express complicated ideas in a chat context while minimizing those two error rates.
226 Chapter 40. Markdown implementation Zulip Documentation, Release 1.4.0
40.4 Zulip’s Changes to Markdown
Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify / disable may already be non-standard.
40.4.1 Basic syntax
• Enable ‘nl2br extension: this means one newline creates a line break (not paragraph break). • Disable italics entirely. This resolves an issue where people were using * and _ and hitting it by mistake too often. E.g. with stock Markdown You should use char * instead of void * there would trigger italics. • Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something) • Add ~~ syntax for strikethrough. • Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having no escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.
40.4.2 Lists
• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it • Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not included in a code block) • Disable ordered list syntax: it automatically renumbers, which can be really confusing when sending a numbered list across multiple messages.
40.4.3 Links
• Enable auto-linkification, both for http://... and guessing at things like t.co/foo. • Force links to be absolute. [foo](google.com) will go to http://google.com, and not http:// zulip.com/google.com which is the default behavior. • Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window • Disable link-by-reference syntax, [foo][bar] ... [bar]: http://google.com • Enable linking to other streams using #**streamName**
40.4.4 Code
• Enable fenced code block extension, with syntax highlighting • Disable line-numbering within fenced code blocks – the