Zulip Documentation Release 1.4.0

1 Zulip overview 3 1.1 Community...... 3 1.2 Installing the Zulip Development environment...... 3 1.3 Running Zulip in production...... 3 1.4 Ways to contribute...... 4 1.5 How to get involved with contributing to Zulip...... 4 1.6 License...... 5

2 Zulip architectural overview 7 2.1 Key Codebases...... 7 2.2 Usage assumptions and concepts...... 7 2.3 Components...... 8

3 Directory structure 11 3.1 Core Python files...... 11 3.2 HTML Templates...... 11 3.3 JavaScript and other static assets...... 12 3.4 Tests...... 12 3.5 Management commands...... 12 3.6 Scripts...... 12 3.7 API and Bots...... 13 3.8 Production puppet configuration...... 13 3.9 Additional Django apps...... 13 3.10 Jinja2 Compatibility Files...... 13 3.11 Translation files...... 13 3.12 Documentation...... 14

4 Zulip 2016 Roadmap 15 4.1 Introduction...... 15 4.2 Burning problems...... 15 4.3 Core User Experience...... 16 4.4 Ease of setup and onboarding issues...... 16 4.5 Internationalization...... 16 4.6 User Experience at scale...... 16 4.7 Administration and management...... 17 4.8 Scalability...... 17 4.9 Performance...... 17 4.10 Technology improvements...... 17

i 4.11 Technical Debt...... 18 4.12 Deployment and upgrade process...... 18 4.13 Security...... 18 4.14 Testing...... 18 4.15 Development environment...... 19 4.16 Documentation...... 19 4.17 Integrations...... 19 4.18 Android app...... 19 4.19 iOS app...... 20 4.20 Desktop apps...... 20 4.21 Community...... 20

5 Version History 21 5.1 Unreleased...... 21 5.2 1.4.2 - 2016-09-27...... 21 5.3 1.4.1 - 2016-09-03...... 21 5.4 1.4 - 2016-08-25...... 21 5.5 1.3.13 - 2016-06-21...... 24 5.6 1.3.12 - 2016-05-10...... 25 5.7 1.3.11 - 2016-05-02...... 25 5.8 1.3.10 - 2016-01-21...... 26 5.9 1.3.9 - 2015-11-16...... 26 5.10 1.3.8 - 2015-11-15...... 26 5.11 1.3.7 - 2015-10-19...... 27

7 Installation 31 7.1 Step 0: Subscribe...... 31 7.2 Step 1: Install SSL Certiﬁcates...... 31 7.3 Step 2: Download and install latest release...... 31 7.4 Step 3: Conﬁgure Zulip...... 32 7.5 Step 4: Initialize Zulip database...... 32 7.6 Step 5: Create a Zulip organization and login...... 32 7.7 Troubleshooting...... 33

8 Troubleshooting 35 8.1 Using supervisorctl...... 35 8.2 Troubleshooting services...... 36

9 Customize Zulip 39 9.1 Integrations...... 39 9.2 Streams and Topics...... 39 9.3 Notiﬁcation settings...... 40 9.4 Mobile and desktop apps...... 40 9.5 All other features...... 40 9.6 Enjoy your Zulip installation!...... 40

10 Secure, maintain, and upgrade 41 10.1 Upgrading...... 41 10.2 Upgrading from a git repository...... 43 10.3 Backups...... 43 10.4 Monitoring...... 44 ii 10.5 Scalability...... 45 10.6 Security Model...... 46 10.7 Management commands...... 48

11 Authentication methods 51 11.1 Adding additional methods using python-social-auth...... 51 11.2 Remote User SSO Authentication...... 51

12 Postgres database details 55 12.1 Remote Postgres database...... 55 12.2 Debugging postgres database issues...... 56 12.3 Stopping the Zulip postgres database...... 56 12.4 Debugging issues starting postgres...... 56 12.5 Postgres Vacuuming alerts...... 57

14 Vagrant environment setup tutorial 61 14.1 Requirements...... 61 14.2 Step 1: Install Prerequisites...... 62 14.3 Step 2: Get Zulip Code...... 64 14.4 Step 3: Start the dev environment...... 64 14.5 Step 4: Developing...... 66 14.6 Troubleshooting & Common Errors...... 68

17 Installing manually on UNIX 77 17.1 On Debian or Ubuntu systems:...... 77 17.2 On Fedora 22 (experimental):...... 78 17.3 On CentOS 7 Core (experimental):...... 79 17.4 On OpenBSD 5.8 (experimental):...... 80 17.5 Common to Fedora/CentOS instructions...... 80 17.6 All Systems:...... 81

20 Writing a new integration 87 20.1 Types of integrations...... 87 20.2 General advice...... 88 20.3 Webhook integrations...... 88 20.4 Python script and plugin integrations...... 89 20.5 Documenting your integration...... 90 20.6 Hello World webhook Walkthrough...... 90

21 Writing a new application feature 97 21.1 General Process in brief...... 97 21.2 Example Feature...... 98

iii 22.2 What is a view?...... 103 22.3 Modifying urls.py...... 103 22.4 Writing human-readable views...... 103 22.5 Writing API REST endpoints...... 104 22.6 Legacy endpoints used by the web client...... 108 22.7 Webhook integration endpoints...... 108

23 Life of a Request 109 23.1 A request is sent to the server, and handled by Nginx...... 109 23.2 Nginx secures traffic with SSL...... 109 23.3 Static files are served directly by Nginx...... 109 23.4 Nginx routes other requests between tornado and django...... 109 23.5 Django routes the request to a view in urls.py files...... 110 23.6 Views serving HTML are internationalized by server path...... 110 23.7 API endpoints use REST...... 110 23.8 Django calls rest_dispatch for REST endpoints, and authenticates...... 111 23.9 The view will authorize the user, extract request variables, and validate them...... 112 23.10 Results are given as JSON...... 112

24 Version control 113 24.1 Commit Discipline...... 113 24.2 Commit Messages...... 114

25 Code style and conventions 115 25.1 Be consistent!...... 115 25.2 Lint tools...... 115 25.3 Secrets...... 115 25.4 Dangerous constructs...... 116 25.5 JS array/object manipulation...... 117 25.6 More arbitrary style things...... 118

26 Testing and writing tests 121 26.1 Overview...... 121 26.2 Running tests...... 121 26.3 Schema and initial data changes...... 122 26.4 Wiping the test databases...... 122 26.5 Manual testing (local app + web browser)...... 123 26.6 Python 3 Compatibility...... 123

27 Python static type checker (mypy) 125 27.1 type_debug.py ...... 125 27.2 Zulip goals...... 126 27.3 Installing mypy...... 126 27.4 Running mypy on Zulip’s code locally...... 126 27.5 Excluded ﬁles...... 127 27.6 Mypy is there to ﬁnd bugs in Zulip before they impact users...... 127 27.7 Annotating strings...... 127

28 Settings system 129 28.1 Server settings...... 129 28.2 Realm settings...... 130

29 Queue processors 133 29.1 Adding a new queue processor...... 133 29.2 Publishing events into a queue...... 134 iv 29.3 Clearing a RabbitMQ queue...... 134

30 Unread counts and the pointer 135 30.1 Pointer logic...... 135 30.2 Unread count logic...... 136

31 Markdown implementation 137 31.1 Zulip’s Markdown philosophy...... 137 31.2 Zulip’s Changes to Markdown...... 138

32 Static asset pipeline 141 32.1 Primary build process...... 141 32.2 Webpack/CommonJS modules...... 141 32.3 Adding static ﬁles...... 141

34 HTML and CSS 145 34.1 Zulip CSS organization...... 145 34.2 Editing Zulip CSS...... 145 34.3 CSS Style guidelines...... 146 34.4 Validating CSS...... 146

35 Full-text search 147 35.1 The default full-text search implementation...... 147 35.2 An optional full-text search implementation...... 147

36 Translating Zulip 149 36.1 Setting Default Language in Zulip...... 149 36.2 Translation Resource Files...... 149 36.3 Transifex Conﬁg...... 149 36.4 Translation Process...... 149 36.5 Backend Translations...... 150 36.6 Frontend Translations...... 151 36.7 Testing Translations...... 152

Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop ﬁle uploads, image previews, group private messages, audible notiﬁca- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Our code is available at our GitHub repository. This set of documents covers installation and contribution instructions. Contents: • Overview • Zulip in production • Installation for developers • Developer tutorials • Code contribution guide • Subsystem documentation Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved | License

There are several places online where folks discuss Zulip. One of those places is our public Zulip instance. You can go through the simple signup process at that link, and then you will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luck ﬁnding core developers roughly between 16:00 UTC and 23:59 UTC. Most questions get answered within a day. We have a Google mailing list that is currently pretty low trafﬁc. It is where we do things like announce public meetings or major releases. You can also use it to ask questions about features or possible bugs. Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a Github account should be able to create Issues there pertaining to bugs or enhancement requests. We also use Pull Requests as our primary mechanism to receive code contributions.

The Zulip development environment is the recommended option for folks interested in trying out Zulip. This is documented in the developer installation guide.

Zulip in production only supports Ubuntu 14.04 right now, but work is ongoing on adding support for additional platforms. The installation process is documented at https://zulip.org/server.html and in more detail in the documentation.

Zulip welcomes all forms of contributions! The page documents the Zulip development process. • Pull requests. Before a pull request can be merged, you need to to sign the Dropbox Contributor License Agreement. Also, please skim our commit message style guidelines. • Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also run them all in your development environment following the instructions in the testing docs. You can also try out our new desktop client, which is in alpha; we’d appreciate testing and feedback. • Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs. Recommended reading for new contributors includes the directory structure and new feature tutorial. You can also improve Zulip.org. • Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . There are also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas where you’d love to see more documentation to the relevant list! Please report any security issues you discover to [email protected]. • App codebases. This repository is for the Zulip server and web app (including most integrations); the desktop, Android, and iOS apps, are separate repositories, as are our experimental React Native iOS app and alpha Electron desktop app. • Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine, and Trello), plus node.js API bindings, and a full-text search PostgreSQL extension, as separate repos. • Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to our translations. See our translating documentation if you’re interested in contributing!

First, subscribe to the Zulip development discussion mailing list. The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have your own project idea in mind or want to get some experience with working on Zulip before embarking on a larger project you have in mind: • Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writing an integration can be a great first contribution. There’s detailed documentation on how to write integrations in the Zulip integration writing guide. • Bite Size: Smaller projects that might be a great first contribution. • Documentation: The Zulip project loves contributions of new documentation. • Help Wanted: A broader list of projects that nobody is currently working on. • Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms. • Bugs: Open bugs. • Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip users are excited about. • 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’re looking to make an impact.

If you’re excited about helping with an open issue, just post on the conversation thread that you’re working on it. You’re encouraged to ask questions on how to best implement or debug your changes – the Zulip maintainers are excited to answer questions to help you stay unblocked and working efficiently. We also welcome suggestions of features that you feel would be valuable or changes that you feel would make Zulip a better open source project, and are happy to support you in adding new features or other user experience improvements to Zulip. If you have a new feature you’d like to add, we recommend you start by opening a GitHub issue about the feature idea explaining the problem that you’re hoping to solve and that you’re excited to work on it. A Zulip maintainer will usually reply within a day with feedback on the idea, notes on any important issues or concerns, and and often tips on how to implement or test it. Please feel free to ping the thread if you don’t hear a response from the maintainers – we try to be very responsive so this usually means we missed your message. For significant changes to the visual design, user experience, data model, or architecture, we highly recommend posting a mockup, screenshot, or description of what you have in mind to zulip-devel@ to get broad feedback before you spend too much time on implementation details. Finally, before implementing a larger feature, we highly recommend looking at the new feature tutorial and coding style guidelines on ReadTheDocs. Feedback on how to make this development process more efficient, fun, and friendly to new contributors is very welcome! Just send an email to the Zulip Developers list with your thoughts.

Copyright 2011-2015 Dropbox, Inc. Licensed under the Apache License, Version 2.0 (the “License”); you may not use this ﬁle except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the speciﬁc language governing permissions and limitations under the License. The software includes some works released by third parties under other free and open source licenses. Those works are redistributed under the license terms under which the works were received. For more details, see the THIRDPARTY ﬁle included with this distribution.

The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon to also support Python 3) and using the Django framework. That codebase includes server-side code and the web client, as well as Python API bindings and most of our integrations with other services and applications (see the directory structure guide). We maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations with Phabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL extension. Our mobile clients are separate code repositories: Android, iOS (stable), and our experimental React Native iOS app. Our legacy desktop application (implemented in QT/WebKit) and our new, alpha cross-platform desktop app (implemented in Electron) are also separate repositories. We use Transifex to do translations. In this overview we’ll mainly discuss the core Zulip server and web application.

Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a small team to more than a thousand users. It features real-time notiﬁcations, message persistence and search, public group conversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, team presence/buddy list, a rich API, Markdown message support, and numerous integrations with other services. The maintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and Mac OS X clients, as well as people using modern web browsers or dedicated Zulip API clients. A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber with its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip realms. The administrators of a realm can choose whether to allow anyone to register an account and join, or only allow people who have been invited, or restrict registrations to members of particular groups (using email domain names or corporate single-sign-on login for veriﬁcation). For more on scalability and security considerations, see the security section of the production maintenance instructions. The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldest message that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). The home screen displays the most recent messages in all the streams a user has joined (except for the streams they’ve muted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the

7 Zulip Documentation, Release 1.4.0 messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow has its own URL. Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming information flow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they find irrelevant.

We use both the Tornado and Django Python web frameworks. Django is the main web application server; Tornado runs the server-to-client real-time push system. The app servers are configured by the Supervisor configuration (which explains how to start the server processes; see “Supervisor” below) and the nginx configuration (which explains which HTTP requests get sent to which app server). Tornado is an asynchronous server and is meant specifically to hold open tens of thousands of long-lived (long-polling or websocket) connections – that is to say, routes that maintain a persistent connection from every running client. For this reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in Tornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip very much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those blocking requests carry a very high performance penalty for a single-threaded, asynchronous server. The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Django application server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on the code path for sending messages.

2.3.2 nginx nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django and Tornado. It handles HTTP requests according to the rules laid down in the many config files found in zulip/puppet/zulip/files/nginx/. zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of these files. It explains what happens when requests come in from outside. • In production, all requests to URLs beginning with /static/ are served from the corresponding files in /home/zulip/prod-static/, and the production build process (tools/build-release-tarball) compiles, minifies, and installs the static assets into the prod-static/ tree form. In development, files are served directly from /static/ in the git repository. • Requests to /json/get_events, /api/v1/events, and /sockjs are sent to the Tornado server. These are requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connection with Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server via puppet/zulip/files/nginx/zulip-include-frontend/upstreams. • Requests to all other paths are sent to the Django app via the UNIX socket unix:/home/zulip/deployments/fastcgi-socket (defined in puppet/zulip/files/nginx/zulip-include-frontend/upstreams). We use zproject/wsgi.py to implement FastCGI here (see django.core.wsgi).

We use supervisord to start server processes, restart them automatically if they crash, and direct logging. The conﬁg ﬁle is zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf. This is where Tor- nado and Django are set up, as well as a number of background processes that process event queues. We use event queues for the kinds of tasks that are best run in the background because they are expensive (in terms of performance) and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing guide.

2.3.4 memcached memcached is used to cache database model objects. zerver/lib/cache.py and zerver/lib/cache_helpers.py manage putting things into memcached, and invalidating the cache when values change. The memcached conﬁguration is in puppet/zulip/files/memcached.conf.

Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, a per-user rate limiting scheme example), and the email-to-Zulip integration. Redis is conﬁgured in zulip/puppet/zulip/files/redis and it’s a pretty standard conﬁguration except for the last line, which turns off persistence:

# Zulip-specific configuration: disable saving to disk. save"" memcached was used ﬁrst and then we added Redis speciﬁcally to implement rate limiting. We’re discussing switching everything over to Redis.

RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initial configuration happens in zulip/scripts/setup/configure-rabbitmq. We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, some analytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used for communication between the application server and the Tornado push system. Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/server/lib/queue.py. There’s an asynchronous client for use in Tornado and a more general client for use elsewhere. zerver/lib/event_queue.py has helper functions for putting events into one queue or another. Most of the processes started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle them. Also see the queuing guide.

PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to live beyond a user’s current session.

In production, Postgres is installed with a default configuration. The directory that would contain configuration files (puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Post- gresql extension. In a development environment, configuration of that postgresql extension is handled by tools/postgres-init-dev-db (invoked by tools/provision.py). That file also manages setting up the development postgresql user. tools/provision.py also invokes tools/do-destroy-rebuild-database to create the actual database with its schema.

Nagios is an optional component used for notiﬁcations to the system administrator, e.g., in case of outages. zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from puppet/zulip/files/nagios_plugins/. This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the Zulip Nagios plugins are intended to be run on the Zulip servers themselves, and are included with the relevant component of the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under /usr/lib/nagios/plugins/zulip_postgres_common).

This page documents the Zulip directory structure, where to find things, and how to decide where to put a file. You may also find the new application feature tutorial helpful for understanding the flow through these files.

Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers. • zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions or templates. • zerver/models.py Main Django models file. Defines Zulip’s database tables. • zerver/lib/actions.py Most code doing writes to user-facing database tables. • zerver/views/*.py Most Django views. • zerver/views/webhooks/ Webhook views for Zulip integrations. • zerver/tornadoviews.py Tornado views. • zerver/worker/queue_processors.py Queue workers. • zerver/lib/*.py Most library code. • zerver/lib/bugdown/ Backend Markdown processor. • zproject/backends.py Authentication backends.

See our translating docs for details on Zulip’s templating systems. • templates/zerver/ For Jinja2 templates for the backend (for zerver app). • static/templates/ Handlebars templates for the frontend.

• static/js/ Zulip’s own JavaScript. • static/styles/ Zulip’s own CSS. • static/images/ Zulip’s images. • static/third/ Third-party JavaScript and CSS that has been vendored. • node_modules/ Third-party JavaScript installed via npm. • assets/ For assets not to be served to the web (e.g. the system to generate our favicons).

• zerver/tests/ Backend tests. • frontend_tests/node_tests/ Node Frontend unit tests. • frontend_tests/casper_tests/ Casper frontend tests. • tools/test-* Developer-facing test runner scripts.

These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access). • zerver/management/commands/ Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly).

• scripts/ Scripts that production deployments might run manually (e.g., restart-server). • scripts/lib/ Scripts that are needed on production deployments but humans should never run directly. • scripts/setup/ Scripts that production deployments will only run once, during installation. • tools/ Scripts used only in a Zulip development environment. These are not included in production release tarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production accidentally (e.g. things that delete the Zulip database without prompting). • tools/setup/ Subdirectory of tools/ for things only used during the development environment setup process. • tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actu- ally test suites should go in tools/.

• api/ Zulip’s Python API bindings (released separately). • api/examples/ API examples. • api/integrations/ Bots distributed as part of the Zulip API bundle. • bots/ Previously Zulip internal bots. These usually need a bit of work.

This is used to deploy essentially all conﬁguration in production. • puppet/zulip/ For conﬁguration for production deployments. • puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.

• confirmation Email conﬁrmation system. • analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). • corporate The old Zulip.com website. Not included in production distribution. • zilencer Primarily used to hold management commands that aren’t used in production. Not included in production distribution.

• zproject/jinja2/__init__.py Jinja2 environment. • zproject/jinja2/backends.py Jinja2 backend. • zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.

• locale/ Backend (Django) translations data ﬁles. • static/locale/ Frontend translations data ﬁles.

You can consult the repository’s .gitattributes ﬁle to see exactly which components are excluded from production releases (release tarballs are generated using tools/build-release-tarball).

Zulip has received a great deal of interest and attention since it was released as free and open source software by Dropbox. That attention has come with a lot of active development work from members of the Zulip community. From when Zulip was released as open source in late September 2015 through today (mid-April, 2016), over 300 pull requests have been submitted to the various Zulip repositories (and over 250 have been merged!), the vast majority of which are submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges the pull requests). In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where the project is hoping to improve. This can be especially important in an open source project like Zulip where development is distributed across many people around the world. This roadmap is intended to organize a list of the most important improvements that should be made to Zulip in the relatively near future. Our aim is to complete most of these improvements in 2016. This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will be used by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product, hold ourselves accountable for making Zulip improve rapidly, and celebrate members of the community who contribute to projects on the roadmap. If you’re someone interested in making a larger contribution to Zulip and looking for somewhere to start, this roadmap is the best place to look for substantial projects that will deﬁnitely be of value to the community (if you’re looking for a starter project, see the guide to getting involved with Zulip). We occasionally update this roadmap by adding strikethrough for issues that have been resolved. Without further ado, below is the Zulip 2016 roadmap.

The top problem for the Zulip project is the state of the mobile apps. The Android app has started seeing rapid progress thanks to a series of contributions by Lisa Neigut of Recurse Center, and we believe it is on a good path. The iOS app has fewer features than Android and has more bugs, but more importantly is in need of an experienced iOS developer who has time to drive the project. Update: Neeraj Wahi is leading an effort on to write a new React Native iOS app for Zulip to replace the old iOS app.

This category includes important improvements to the core user experience that will benefit all users. • Improve missed message notifications to make “reply” work nicely • Add support for showing “user is typing” notifications • Add pretty bubbles for recipients in the compose box • Finish and merge support for pinning a few important streams • Display stream descriptions more prominently • Integration inline URL previews • Add support for managing uploaded files • Make Zulip onboarding experience smoother for teams not used to topics. That specific proposal might not be right but the issue is worth investing time in.

This category focuses on issues users experience when installing a new Zulip server or setting up a new Zulip realm. • Create a web ﬂow for setting up a new realm / the ﬁrst realm on a new server (currently, it’s a command-line process) • Document or better script solution to rabbitmq startup issues • Add a mechanism for deleting early test messages • Merge a supported way to use Zulip in Docker in production implementation.

The core Zulip UI has been mostly translated into 5 languages; however, more work is required to make those translations actually displayed in the Zulip UI for the users who would beneﬁt from them. • Merge support for using translations in Django templates • Add text in handlebars templates to translatable string database • Merge support for translating text in handlebars • Add text in error messages to translatable strings

There are a few parts of the Zulip UI which could beneﬁt from overhauls designed around making the user experience nice for large teams. • Make the buddy list work better for large teams • Improve @-mentioning syntax based on stronger unique identiﬁers • Show subscriber counts on streams

• Make the streams page easier to navigate with 100s of streams • Add support for ﬁltering long lists of streams

Currently, Zulip has a number of administration features that can be controlled only via the command line. • Make default streams web-configurable • Make realm emoji web-configurable • Make realm filters web-configurable • Make realm aliases web-configurable • Enhance the LDAP integration and make it web-configurable • Add a SAML integration for Zulip • Improve administrative controls for managing streams

Zulip should support 10000 users in a realm and also support smaller realms in more resource-constrained environments (probably a good initial goal is working well with only 2GB of RAM). • Make the Zulip Tornado service support horizontal scaling • Make presence system scale well to 10000 users in a realm. • Support running queue workers multithreaded in production to decrease minimum memory footprint

Performance is essential for a communication tool. While some things are already quite good (e.g. narrowing and message sending is speedy), this is an area where one can always improve. There are a few known performance opportunities: • Migrate to faster jinja2 templating engine • Don’t load zxcvbn when it isn’t needed • Optimize the frontend performance of loading the Zulip webapp using proﬁling

Zulip should be making use of the best Python/Django tools available. • Add support for Zulip running on Python 3 • Add support for changing users’ email addresses • Automatic thumbnailing of uploaded images

• Upgrade Zulip to use Django 1.10 once it is released. The patches needed to run Zulip were merged into mainline Django in Django 1.10, so this will mean we don’t need to use a fork of Django anymore.

While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way. • Migrate most web routes to REST API • Finish purging global variables from the Zulip JavaScript • Finish deprecating and remove the pre-REST Zulip /send_message API • Split Tornado subsystem into a separate Django app • Clean up clutter in the root of the zulip.git repository • Refactor zulip.css to be broken into components

• Support backwards-incompatible upgrades to Python libraries • Minimize the downtime required in the Zulip upgrade process

• Add support for 2-factor authentication on all platforms • Add a retention policy feature that automatically deletes old messages • Upgrade every Zulip dependency to a modern version • The LOCAL_UPLOADS_DIR ﬁle uploads backend only supports world-readable uploads • Add support for stronger security controls for uploaded ﬁles

• Extend Zulip’s automated test coverage to include all API endpoints • Build automated tests for the client API bindings • Add Python static type-checking to Zulip using mypy • Improve the runtime of Zulip’s backend test suite • Use caching to make Travis CI runtimes faster • Add automated tests for the production upgrade process • Improve Travis CI “production” test suite to catch more regressions

• Migrate from jslint to eslint • Figure out a nice upgrade process for Zulip Vagrant VMs • Replace closure-compiler with a faster miniﬁer toolchain • Add support for building frontend features in React • Use a JavaScript bundler like webpack

• Signiﬁcantly expand documentation of the Zulip API and integrating with Zulip. • Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developers or system administrators. • Expand developer documentation with more tutorials explaining how to do various types of projects. • Overhaul new contributor documentation, especially on coding style, to better highlight and teach the important pieces. • Update all screenshots to show the current Zulip UI

Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhook integrations for getting notiﬁcations into Zulip, it’d be great to streamline that process and make bots that receive messages just as easy to build. • Make it super easy to take screenshots for new webhook integrations • Add an outgoing webhook integration system • Build a framework to cut duplicated code in new webhook integrations • Make setting up a new integration a smooth ﬂow • Optimize the integration writing documentation to make writing new ones really easy.

The Zulip Android app is ahead of the iOS app in terms of feature set, but there is still a lot of work to do. Most of the things listed below will eventually apply to the iOS app as well. • Support using a non-zulip.com server • Support Google authentication with a non-Zulip.com server • Add support for narrowing to @-mentions • Support having multiple Zulip realms open simultaneously • Build a slick development login page to simplify testing (similar to the development homepage on web) • Improve the compose box to let you see what you’re replying to

• Make it easy to compose messages with mentions, emoji, etc. • Display unread counts and improve navigation • Hide messages sent to muted topics • Fill out documentation to make it easy to get started

Most of the projects listed under Android apply here as well, but it’s worth highlighting some areas where iOS is substantially behind Android. The top priority here is recruiting a lead developer for the iOS app. Once we have that resolved, we’ll expand our ambitions for the app with more speciﬁc improvements. • iOS app needs maintainer • APNS notiﬁcations are broken

The top goal for the desktop apps is to rebuild it in a modern toolchain so that it’s easy for a wide range of developers to contribute to the apps. The new cross-platform app is implemented in Electron, a framework (maintained by GitHub) that uses Chromium and Node.js, so Zulip developers only need to write HTML, CSS, and JavaScript. The new Zulip app is in alpha as of early August 2016. • Migrate platform from QT/webkit to Electron • Desktop app doesn’t recover well from entering the wrong Zulip server • Support having multiple Zulip realms open simultaneously • Build an efﬁcient process for testing and releasing new versions of the desktop apps

These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project. • Setup a Zulip server for the Zulip development community • Expand the number of core developers able to do code reviews • Expand the number of contributors regularly adding features to Zulip • Have a successful summer with Zulip’s 3 GSOC students

• Upgraded Django to version 1.8.15 (with the Zulip patches applied), ﬁxing a CSRF vulnerability in Django (see https://www.djangoproject.com/weblog/2016/sep/26/security-releases/), and a number of other Django bugs from past Django stable releases that largely affects parts of Django that are not used by Zulip. • Fixed buggy logrotate conﬁguration.

• Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0. • Fixed local ﬁle uploads integration being broken for new 1.4.0 installations.

• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a major change to how Zulip is installed that we expect will simplify upgrades in the future. • Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached network transfer for loading Zulip. • Added support for using Ubuntu Xenial in production. • Added a powerful and complete realm import/export tool. • Added nice UI for selecting a default language to display settings. • Added UI for searching streams in left sidebar with hotkeys. • Added Semaphore, Bitbucket, and HelloWorld (example) integrations. • Added new webhook-based integration for Trello.

• Added management command for creating realms through web UI. • Added management command to send password reset emails. • Added endpoint for mobile apps to query available auth backends. • Added LetsEncrypt documentation for getting SSL certificates. • Added nice rendering of unicode emoji. • Added support for pinning streams to the top of the left sidebar. • Added search box for filtering user list when creating a new stream. • Added realm setting to disable message editing. • Added realm setting to time-limit message editing. Default is 10m. • Added realm setting for default language. • Added year to timestamps in message interstitials for old messages. • Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authentication methods). • Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulip server. • Added numerous hooks to puppet modules to enable more configurations. • Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.). • Added automatic configuration of postgres/memcached settings based on the server’s available RAM. • Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo. • Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4. • Added support for Name format when inviting users. • Added numerous special-purpose settings options. • Added a hex input field in color picker. • Documented new Electron beta app and mobile apps in the /apps/ page. • Enabled Android Google authentication support. • Enhanced logic for tracking origin of user uploads. • Improved error messages for various empty narrows. • Improved missed message emails to better support directly replying. • Increased backend test coverage of Python code to 85.5%. • Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properly handle unicode. • Fixed major i18n-related frontend performance regression on /#subscriptions page. Saves several seconds of load time with 1k streams. • Fixed Jinja2 migration bug when trying to register an email that already has an account. • Fixed narrowing to a stream from other pages. • Fixed various frontend strings that weren’t marked for translation. • Fixed several bugs around editing status (/me) messages.

• Fixed queue workers not restarting after changes in development. • Fixed Casper tests hanging while development server is running. • Fixed browser autocomplete issue when adding new stream members. • Fixed broken create_stream and rename_stream management commands. • Fixed zulip-puppet-apply exit code when puppet throws errors. • Fixed EPMD restart being attempted on every puppet apply. • Fixed message cache filling; should improve perf after server restart. • Fixed caching race condition when changing user objects. • Fixed buggy puppet configuration for supervisord restarts. • Fixed some error handling race conditions when editing messages. • Fixed fastcgi_params to protect against the httpoxy attack. • Fixed bug preventing users with mit.edu emails from registering accounts. • Fixed incorrect settings docs for the email mirror. • Fixed APNS push notification support (had been broken by Apple changing the APNS API). • Fixed some logic bugs in how attachments are tracked. • Fixed unnecessarily resource-intensive rabbitmq cron checks. • Fixed old deployment directories leaking indefinitely. • Fixed need to manually add localhost in ALLOWED_HOSTS. • Fixed display positioning for the color picker on subscriptions page. • Fixed escaping of Zulip extensions to markdown. • Fixed requiring a reload to see newly uploaded avatars. • Fixed @all warning firing even for @all. • Restyled password reset form to look nice. • Improved formatting in reset password links. • Improved alert words UI to match style of other settings. • Improved error experience when sending to nonexistent users. • Portions of integrations documentation are now automatically generated. • Restructured the URLs files to be more readable. • Upgraded almost all Python dependencies to current versions. • Substantially expanded and reorganized developer documentation. • Reorganized production documentation and moved to ReadTheDocs. • Reorganized .gitignore type files to be written under var/ • Refactored substantial portions of templates to support subdomains. • Renamed local_settings.py symlink to prod_settings.py for clarity. • Renamed email-mirror management command to email_mirror. • Changed HTTP verb for create_user_backend to PUT.

• Eliminated all remaining settings hardcoded for zulip.com. • Eliminated essentially all remaining hardcoding of mit.edu. • Optimized the performance of all the test suites. • Optimized Django memcached configuration. • Removed old prototype data export tool. • Disabled insecure RC4 cipher in nginx configuration. • Enabled shared SSL session cache in nginx configuration. • Updated header for Zulip static assets to reflect Zulip being open source.

• Added nearly complete internationalization of the Zulip UI. • Added warning when using @all/@everyone. • Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to. • Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io. • Added menu option to mark all messages in a stream or topic as read. • Added new Attachment model to keep track of uploaded ﬁles. • Added caching of virtualenvs in development. • Added mypy static type annotations to about 85% of the Zulip Python codebase. • Added automated test of backend templates to test for regressions. • Added lots of detailed documentation on the Zulip development environment. • Added setting allowing only administrators to create new streams. • Added button to exit the Zulip tutorial early. • Added web UI for conﬁguring default streams. • Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms on a Zulip server. • Fixed email_gateway_password secret not working properly. • Fixed missing helper scripts for RabbitMQ Nagios plugins. • Fixed skipping forward to latest messages (“More messages below” button). • Fixed netcat issue causing Zulip installation to hang on Scaleway machines. • Fixed rendering of /me status messages after message editing. • Fixed case sensitivity of right sidebar fading when compose is open. • Fixed error messages when composing to invalid PM recipients. • Fixed LDAP auth backend not working with Zulip mobile apps. • Fixed erroneous WWW-Authenticate headers with expired sessions. • Changed “coworkers” to “users” in the Zulip UI. • Changed add_default_stream REST API to correctly use PUT rather than PATCH.

• Updated the Zulip emoji set (the Android Emoji) to a modern version. • Made numerous small improvements to the Zulip development experience. • Migrated backend templates to the faster Jinja2 templating system. • Migrated development environment setup scripts to tools/setup/. • Expanded test coverage for several areas of the product. • Simpliﬁed the API for writing new webhook integrations. • Removed most of the remaining JavaScript global variables.

• CVE-2016-4426: Bot API keys were accessible to other users in the same realm. • CVE-2016-4427: Deactivated users could access messages if SSO was enabled. • Fixed a RabbitMQ conﬁguration bug that resulted in reordered messages. • Added expansive test suite for authentication backends and decorators. • Added an option to logout_all_users to delete only sessions for deactivated users.

• Moved email digest support into the default Zulip production configuration. • Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py. • Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly (e.g. Google Hangouts). • Added new management command to test sending email from Zulip. • Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations. • Added Nagios plugins to the main distribution. • Added ability for realm administrators to manage custom emoji. • Added guide to writing new integrations. • Enabled camo image proxy to fix mixed-content warnings for http images. • Refactored the Zulip puppet modules to be more modular. • Refactored the Tornado event system, fixing old memory leaks. • Removed many old-style /json API endpoints • Implemented running queue processors multithreaded in development, decreasing RAM requirements for a Zulip development environment from ~1GB to ~300MB. • Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant performance issue in larger realms. • Fixed the disabling of desktop notifications from 1.3.7 for new users. • Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for the realm.

• Fixed bugs changing certain settings in administration pages. • Fixed collapsing messages in narrowed views. • Fixed 500 errors when uploading a non-image ﬁle as an avatar. • Fixed Jira integration incorrectly not @-mentioning assignee.

• Added new integration for Travis CI. • Added settings option to control maximum file upload size. • Added support for running Zulip development environment in Docker. • Added easy configuration support for a remote postgres database. • Added extensive documentation on scalability, backups, and security. • Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature. • Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf. • Improved the styling for the Administration page and added tabs. • Substantially improved loading performance on slow networks by enabling GZIP compression on more assets. • Changed the page title in narrowed views to include the current narrow. • Fixed several backend performance issues affecting very large realms. • Fixed bugs where draft compose content might be lost when reloading site. • Fixed support for disabling the “zulip” notifications stream. • Fixed missing step in postfix_localmail installation instructions. • Fixed several bugs/inconveniences in the production upgrade process. • Fixed realm restrictions for servers with a unique, open realm. • Substantially cleaned up console logging from run-dev.py.

• Added options to the Python api for working with untrusted server certiﬁcates. • Added a lot of documentation on the development environment and testing. • Added partial support for translating the Zulip UI. • Migrated installing Node dependencies to use npm. • Fixed LDAP integration breaking autocomplete of @-mentions. • Fixed admin panel reactivation/deactivation of bots.

• Fixed inaccurate documentation for downloading the desktop apps. • Fixed various minor bugs in production installation process. • Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) who were invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230). • Major preliminary progress towards supporting Python 3.

• Turn off desktop and audible notiﬁcations for streams by default. • Added support for the LDAP authentication integration creating new users. • Added new endpoint to support Google auth on mobile. • Fixed desktop notiﬁcations in modern Firefox. • Fixed several installation issues for both production and development environments. • Improved documentation for outgoing SMTP and the email mirror integration.

Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a development environment following these instructions, since then you don’t need to worry about setting up SSL certiﬁcates and an authentication mechanism. Or, you can check out the developers’ chatroom (a public, running Zulip instance).

• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM. For installations with fewer users, 1 CPU and 2GB RAM might be sufﬁcient. We strong recommend against installing with less than 2GB of RAM, as you will likely experience out of memory issues. • Disk space: You’ll need at least 10GB of free disk space. If you intend to store uploaded ﬁles locally rather than on S3 you will likely need more.

• Outgoing HTTP(S) access to the public Internet. If you want to be able to send email from Zulip, you’ll also need SMTP access.

Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running Zulip in production. 64-bit is recommended.

You should already have a domain name available for your Zulip production instance. In order to generate valid SSL certiﬁcates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update the domains A record to point to your production server.

• SSL Certiﬁcate for the host you’re putting this on (e.g. zulip.example.com). The installation instructions contain documentation for how to get an SSL certiﬁcate for free using LetsEncrypt.

• Email credentials Zulip can use to send outgoing emails to users (e.g. email address conﬁrmation emails during the signup process, missed message notiﬁcations, password reminders if you’re not using SSO, etc.). Once you have met these requirements, see full instructions for installing Zulip in production.

Ensure you have an Ubuntu system that satisﬁes the installation requirements. In short, you should have an Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server instance, with at least 4GB RAM, 2 CPUs, and 10 GB disk space. You should also have a domain name available and have updated its DNS record to point to the server.

Please subscribe to low-trafﬁc the Zulip announcements Google Group to get announcements about new releases, security issues, etc.

Zulip runs over https only and requires ssl certificates in order to work. It looks for the certificates in /etc/ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note that Zulip uses nginx as its webserver and thus expects a chained certificate bundle If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, just install (or symlink) them into place at the above paths, and move on to the next step.

If you haven’t already, download and unpack the latest built server tarball with the following commands: sudo-i # If not already root wget https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz rm-rf/root/zulip&& mkdir/root/zulip tar-xf zulip-server-latest.tar.gz--directory=/root/zulip--strip-components=1

This may take a while to run, since it will install a large number of dependencies. The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue that caused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; please include a copy of that ﬁle in any bug reports.

Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the mandatory settings, which are all found under the heading ### MANDATORY SETTINGS. These settings include: • EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation. This will be the domain for which you have DNS A records pointing to this server and for which you configured SSL certificates. • ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who will get support emails. • AUTHENTICATION_BACKENDS: a list of enabled authentication mechanisms. You’ll need to enable at least one authentication mechanism by uncommenting its corresponding line, and then also do any additional configuration required for that backend as documented in the settings.py file. See the section on Authentication for more detail on the available authentication backends and how to configure them. • EMAIL_*, DEFAULT_FROM_EMAIL, and NOREPLY_EMAIL_ADDRESS: Regardless of which authentication backends you enable, you must provide settings for an outgoing SMTP server so Zulip can send emails when needed. We highly recommend testing your configuration using manage.py send_test_email to confirm your outgoing email configuration is working correctly. • ALLOWED_HOSTS: Replace * with the fully qualified DNS name for your Zulip server here.

At this point, you are done doing things as root. To initialize the Zulip database for your production install, run:

The initialize-database script will report an error if you did not ﬁll in all the mandatory settings from /etc/zulip/settings.py. It is safe to rerun it after correcting the problem if that happens. This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create an organization in your Zulip installation.

• If you haven’t already, verify that your server can send email using ./manage.py send_test_email [email protected]. You’ll need working outgoing email to complete the setup process. • Run the organization (realm) creation management command :

su zulip # If you weren't already the zulip user cd/home/zulip/deployments/current ./manage.py generate_realm_creation_link

This will print out a secure 1-time use link that allows creation of a new Zulip organization on your server. For most servers, you will only ever do this once, but you can run manage.py generate_realm_creation_link again if you want to host another organization on your Zulip server. • Open the link generated with your web browser. You’ll see the create organization page (screenshot here). Enter your email address and click Create organization.

• Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up your organization and initial administrator user (screenshot here). Complete this form and log in! Congratulations! You are logged in as an organization administrator for your new Zulip organization. After getting oriented, we recommend visiting the special “Administration” tab linked to from the upper-right gear menu in the Zulip app to configure important policy settings like how users can join your new organization. By default, your organization will be configured as follows (screenshot here): • restricted_to_domain=True: Only people with emails with the same ending as yours can join. • invite_required=False: An invitation is not required to join the realm. • invite_by_admin_only=False: You don’t need to be an admin user to invite other users. Next, you’ll likely want to do one of the following: • Customize your Zulip organization. • Learn about managing a production Zulip server.

If you get an error after scripts/setup/install completes, check /var/log/zulip/errors.log for a traceback, and consult the troubleshooting section for advice on how to debug. If that doesn’t help, please visit the “installation help” stream in the Zulip developers’ chat for realtime help or email [email protected] with the traceback and we’ll try to help you out!

Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, to learn how to use the Supervisor client to monitor and manage services. If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Components section. This will help you understand the many services Zulip uses. If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/. That directory contains one log ﬁle for each service, plus errors.log (has all errors), server.log (has logs from the Django and Tornado servers), and workers.log (has combined logs from the queue workers). The section troubleshooting services on this page includes details about how to ﬁx common issues with Zulip services. If you run into additional problems, please report them so that we can update this page! The Zulip installation scripts logs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks from that log.

To see what Zulip-related services are conﬁgured to use Supervisor, look at /etc/supervisor/conf.d/zulip.conf and /etc/supervisor/conf.d/zulip-db.conf. Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.

When everything is running as expected, you will see something like this: process-fts-updates RUNNING pid 2194,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-0 RUNNING pid 2209,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-1 RUNNING pid 2210,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-2 RUNNING pid 2211,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-4 RUNNING pid 2208,

˓→uptime1:13:11 zulip-workers:zulip-deliver-enqueued-emails STARTING zulip-workers:zulip-events-confirmation-emails RUNNING pid 2199,

˓→uptime1:13:11 zulip-workers:zulip-events-feedback_messages RUNNING pid 2207,

˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_mobile_notifications RUNNING pid 2204,

˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_reminders RUNNING pid 2206,

˓→uptime1:13:11 zulip-workers:zulip-events-user-activity-interval RUNNING pid 2196,

After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want to restart the Zulip application. You can restart Zulip using: supervisorctl restart all

The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise support the Zulip application: • postgresql • rabbitmq-server

• nginx • redis • memcached If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common conﬁguration problems and how to resolve them: • An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running; to ﬁx this, try:

If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configuration; you can often correct this by configuring /etc/hosts properly. • If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To fix this, configure them properly and then run:

Once you’ve got Zulip setup, you’ll likely want to conﬁgure it the way you like. There are four big things to focus on: 1. Integrations 2. Streams and Topics 3. Notiﬁcation settings 4. Mobile and desktop apps Lastly, read about Zulip’s other great features, and then enjoy your Zulip installation!

We recommend setting up integrations for the major tools that your team works with. For example, if you’re a software development team, you may want to start with integrations for your version control, issue tracker, CI system, and monitoring tools. Spend time configuring these integrations to be how you like them – if an integration is spammy, you may want to change it to not send messages that nobody cares about (E.g. for the zulip.com trac integration, some teams find they only want notifications when new tickets are opened, commented on, or closed, and not every time someone edits the metadata). If Zulip doesn’t have an integration you want, you can add your own! Most integrations are very easy to write, and even more complex integrations usually take less than a day’s work to build. We very much appreciate contributions of new integrations; see the brief integration writing guide. It can often be valuable to integrate your own internal processes to send notifications into Zulip; e.g. notifications of new customer signups, new error reports, or daily reports on the team’s key metrics; this can often spawn discussions in response to the data.

If it feels like a stream has too much trafﬁc about a topic only of interest to some of the subscribers, consider adding or renaming streams until you feel like your team is working productively. Second, most users are not used to topics. It can require a bit of time for everyone to get used to topics and start beneﬁtting from them, but usually once a team is using them well, everyone ends up enthusiastic about how much topics make life easier. Some tips on using topics:

• When replying to an existing conversation thread, just click on the message, or navigate to it with the arrow keys and hit “r” or “enter” to reply on the same topic • When you start a new conversation topic, even if it’s related to the previous conversation, type a new topic in the compose box • You can edit topics to ﬁx a thread that’s already been started, which can be helpful when onboarding new batches of users to the platform. Third, setting default streams for new users is a great way to get new users involved in conversations before they’ve accustomed themselves with joining streams on their own. You can use the set_default_streams command to set default streams for users within a realm: python manage.py set_default_streams--domain=example.com--streams=foo,bar,...

Zulip gives you a great deal of control over which messages trigger desktop notifications; you can configure these extensively in the /#settings page (get there from the gear menu). If you find the desktop notifications annoying, consider changing the settings to only trigger desktop notifications when you receive a PM or are @-mentioned.

Currently, the Zulip Desktop app only supports talking to servers with a properly signed SSL certificate, so you may find that you get a blank screen when you connect to a Zulip server using a self-signed certificate. The Zulip Android app in the Google Play store doesn’t yet support talking to non-zulip.com servers (and the iOS one doesn’t support Google auth SSO against non-zulip.com servers; there’s a design for how to fix that which wouldn’t be a ton of work to implement). If you are interested in helping out with the Zulip mobile apps, shoot an email to [email protected] and the maintainers can guide you on how to help. For announcements about improvements to the apps, make sure to join the [email protected] list so that you can receive the announcements when these become available.

Hotkeys, emoji, search ﬁlters, @-mentions, etc. Zulip has lots of great features, make sure your team knows they exist and how to use them effectively.

If you discover things that you wish had been documented, please contribute documentation suggestions either via a GitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation cover everything anyone might want to know about running Zulip in production. Next: Maintaining and upgrading Zulip in production.

This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including: • Upgrading • Upgrading from a git repository • Backups • Monitoring • Scalability • Security Model • Management commands

We recommend reading this entire section before doing your ﬁrst upgrade. To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.com/ dist/releases/ You also have the option of creating your own release tarballs from a copy of zulip.git repository using tools/build-release-tarball. And, starting with Zulip version 1.4, you can upgrade Zulip to a version in a Git repository directly. Next, run as root:

/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz

The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any database migrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service, which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgrade in advance, we recommend doing upgrades at off hours. Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will require signiﬁcant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5. Contributions on testing and documenting this process are welcome!

Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgrade process will overwrite your configuration when it does the puppet apply. You can test whether this will happen assuming no upstream changes to the configuration using scripts/zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restore your configuration. If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough to handle your setup.

The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs to include output that shows all errors in any bug reports. After the upgrade, we recommend checking /var/log/zulip/errors.log to conﬁrm that your users are not experiencing errors after the upgrade.

The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ containing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/current and /root/zulip as part of the upgrade process. This means that if the new version isn’t working, you can quickly downgrade to the old version by using /home/zulip/deployments//scripts/restart-server to return to a previous version that you’ve deployed (the version is speciﬁed via the path to the copy of restart-server you call).

If required, you can update your settings by editing /etc/zulip/settings.py and then run /home/zulip/deployments/current/scripts/restart-server to restart the server.

While the Zulip upgrade script runs apt-get upgrade, you are responsible for running this on your system on a regular basis between Zulip upgrades to ensure that it is up to date with the latest security patches.

To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g. https://zulip.example.com/api. Our Python API example scripts support this via the --site=https://zulip.example.com argument. The API bindings support it via putting site=https://zulip.example.com in your .zuliprc. Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc ﬁle or the environment), but this is not yet documented for some of the integrations (the included integration documentation on /integrations will properly document how to do this for most integrations). We welcome pull requests for integrations that don’t discuss this!

Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktop and mobile apps.

As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automatically restarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise conﬁguration.

Starting with version 1.4, the Zulip server supports doing deployments from a Git repository. To conﬁgure this, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf ﬁle’s puppet_classes entry, like this:

Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. You can conﬁgure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; by default it uses the main zulip repository (shown below).

Once that is done (and assuming the currently installed version of Zulip is new enough that this script exists), you can do deployments by running as root:

and Zulip will automatically fetch the relevant branch from the speciﬁed repository, build the static assets, and deploy that version. Currently, the upgrade process is slow, but it doesn’t need to be; there is ongoing work on optimizing it.

There are several pieces of data that you might want to back up: • The postgres database. That you can back up like any postgres database; we have some example tooling for doing that incrementally into S3 using wal-e in puppet/zulip_internal/manifests/postgres_common.pp (that’s what we use for zulip.com’s database backups). Note that this module isn’t part of the Zulip server releases since it’s part of the zulip.com configuration (see https://github.com/zulip/zulip/issues/293 for a ticket about fixing this to make life easier for running backups). • Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if you have instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in that directory and you’ll want to back it up. • Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key, all users will need to login again when you setup a replacement server since you won’t be able to verify their cookies; if you lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar filenames are computed using a hash of avatar_salt and user’s email), etc. • The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and it’s definitely lower-priority.

To restore from backups, the process is basically the reverse of the above: • Install new server as normal by downloading a Zulip release tarball and then using scripts/setup/install, you don’t need to run the initialize-database second stage which puts default data into the database. • Unpack to /etc/zulip the settings.py and secrets.conf files from your backups. • Restore your database from the backup using wal-e; if you ran initialize-database anyway above, you’ll want to first scripts/setup/postgres-init-db to drop the initial database first. • If you’re using local file uploads, restore those files to the path specified by settings.LOCAL_UPLOADS_DIR and (if appropriate) any logs. • Start the server using scripts/restart-server This restoration process can also be used to migrate a Zulip installation from one server to another. We recommend running a disaster recovery after you setup backups to confirm that your backups are working; you may also want to monitor that they are up to date using the Nagios plugin at: puppet/zulip_internal/files/nagios_plugins/check_postgres_backup. Contributions to more fully automate this process or make this section of the guide much more explicit and detailed are very welcome!

Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files: • puppet/zulip_internal/manifests/postgres_slave.pp • puppet/zulip_internal/manifests/postgres_master.pp • puppet/zulip_internal/files/postgresql/* Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the main puppet/zulip/ tree) would be very welcome!

The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available under puppet/zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs). The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirec- tories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: • check_send_receive_time (sends a test message through the system between two bot users to check that end-to- end message sending works) • check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or the queue workers being behind) • check_queue_worker_errors (checks for errors reported by the queue workers) • check_worker_memory (monitors for memory leaks in queue workers)

• check_email_deliverer_backlog and check_email_deliverer_process (monitors for whether outgoing emails are being sent) Database monitoring: • check_postgres_replication_lag (checks streaming replication is up to date). • check_postgres (checks the health of the postgres database) • check_postgres_backup (checks backups are up to date; see above) • check_fts_update_log (monitors for whether full-text search updates are being processed) Standard server monitoring: • check_website_response.sh (standard HTTP check) • check_debian_packages (checks apt repository is up to date) If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it in production are encouraged!

This section attempts to address the considerations involved with running Zulip with a large team (>1000 users). • We recommend using a remote postgres database for isolation, though it is not required. In the following, we discuss a relatively simple configuration with two types of servers: application servers (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers. • You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with just a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get away with half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users, you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoid high rates of memcached misses. • For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. We have not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’s m3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The database load per user is pretty optimized as long as memcached is working correctly. This has not been tested, but from extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active users using a single large database server without doing anything complicated like sharding the database. • For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (using Postgres streaming replication; see the section on configuring this). Be sure to check out the section on backups if you’re hoping to run a spare application server; in particular you probably want to use the S3 backend for storing user-uploaded files and avatars and will want to make sure secrets are available on the hot spare. • Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There are two issues: you need to share the memcached/redis/rabbitmq instance (these should can be moved to a network service shared by multiple servers with a bit of configuration) and the Tornado event system for pushing to browsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other. One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes on a single server, which is also likely the first part of any project to support exchanging events amongst multiple servers. Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping to improve.

This section attempts to document the Zulip security model. Since this is new documentation, it likely does not cover every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip development mailing list (or if you think you’ve found a security bug, please report it to [email protected] so we can do a responsible security announcement).

• It’s reasonable to think about security for a Zulip server like you do security for a team email server – only trusted administrators within an organization should have shell access to the server. In particular, anyone with root access to a Zulip application server or Zulip database server, or with access to the zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data (so they can read messages, modify history, etc.). It would be difﬁcult or impossible to avoid this, because the server needs access to the data to support features expected of a group chat system like the ability to search the entire message history, and thus someone with control over the server has access to that data as well.

• Trafﬁc between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. • The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar. Zulip stores user passwords using the standard PBKDF2 algorithm. Password strength is checked and weak passwords are visually discouraged using the zxcvbn library, but Zulip does not by default have strong requirements on user password strength. Modify static/js/common.js to adjust the password strength requirements (Patches welcome to make controlled by an easy setting!). • Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks.

• Zulip message content is rendering using a specialized Markdown parser which escapes content to protect against cross-site scripting attacks. • Zulip supports both public streams and private (“invite-only”) streams. Any Zulip user can join any public stream in the realm (and can view the complete message of any public stream history without joining the stream). • Users who are not members of a private stream cannot read messages on the stream, send messages to the stream, or join the stream, even if they are a Zulip administrator. However, any member of a private stream can invite other users to the stream. When a new user joins a private stream, they can see future messages sent to the stream, but they do not receive access to the stream’s message history. • Zulip supports editing the content or topics of messages that have already been sent (and even updating the topic of messages sent by other users when editing the topic of the overall thread). While edited messages are synced immediately to open browser windows, editing messages is not a safe way to redact secret content (e.g. a password) unintentionally shared via Zulip, because other users may have seen and saved the content of the original message (for example, they could have taken a screenshot immediately after you sent the message, or have an API tool recording all messages they receive). Zulip stores and sends to clients the content of every historical version of a message, so that future versions of Zulip could support displaying the diffs between previous versions.

• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators have the ability to deactivate and reactivate other human and bot users, delete streams, add/remove administrator privileges, as well as change conﬁguration for the overall realm (e.g. whether an invitation is required to join the realm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages or the messages sent to private streams to which the administrator is not subscribed. However, a Zulip administrator subscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators have administrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to access messages sent to private streams that have bots subscribed, by using the bot’s credentials. In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g. to support auditing functionality). • Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. • To properly remove a user’s access to a Zulip team, it does not sufﬁce to change their password or deactivate their account in the SSO system, since neither of those prevents authenticating with the user’s API key or those of bots the user has created. Instead, you should deactivate the user’s account in the Zulip administration interface (/#administration); this will automatically also deactivate any bots the user had created. • The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s API key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone is lost, in addition to changing passwords, you should rotate the user’s Zulip API key. • Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do including reading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). In particular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm. A likely future feature for Zulip is limited bots that can only send messages. • Certain Zulip bots can be marked as “API super users”; these special bots have the ability to send messages that appear to have been sent by another user (an important feature for implementing integrations like the Jabber, IRC, and Zephyr mirrors).

• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 integration. The URLs of user-uploaded files are secret; if you are using the “local file upload” integration, anyone with the URL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtle attack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access to the file might be leaked to the other server via the Referrer header (see https://github.com/zulip/zulip/issues/320). The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented to users don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevant realm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping the URL secret is still important to avoid other users in the Zulip realm from being able to access the file. • Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted into the Zulip message feed by other users over HTTPS. • By default, Zulip will provide image previews inline in the body of messages when a message contains a link to an image. You can control this using the INLINE_IMAGE_PREVIEW setting.

If you ﬁnd some aspect of Zulip that seems inconsistent with this security model, please report it to zulip- [email protected] so that we can investigate and coordinate an appropriate security release if needed. Zulip security announcements will be sent to [email protected], so you should subscribe if you are running Zulip in production.

Zulip has a large library of Django management commands. To use them, you will want to be logged in as the zulip user and for the purposes of this documentation, we assume the current working directory is /home/zulip/deployments/current. Below, we should several useful examples, but there are more than 100 in total. We recommend skimming the usage docs (or if there are none, the code) of a management command before using it, since they are generally less polished and more designed for expert use than the rest of the Zulip system.

You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g. you can do the following to change an email address:

$ /home/zulip/deployments/current/manage.py shell In [1]: user_profile = get_user_profile_by_email("[email protected]") In [2]: do_change_user_email(user_profile, "[email protected]") manage.py dbshell

You can make any user a realm administrator on the command line with the knight management command:

If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create api super users. To do this, use ./manage.py knight with the --permission=api_super_user argument. See bots/irc-mirror.py and bots/jabber_mirror.py for further detail on these.

There are a large number of useful management commands under zerver/manangement/commands/; you can also see them listed using ./manage.py with no arguments.

One such command worth highlighting because it’s a valuable feature with no UI in the Administration page is ./manage.py realm_filters, which allows you to configure certain patterns in messages to be automatically linkified, e.g., whenever someone mentions “T1234”, it could be auto-linkified to ticket 1234 in your team’s Trac instance. Next: Remote User SSO Authentication.

Zulip supports several different authentications methods: • EmailAuthBackend - Email/password authentication. • ZulipLDAPAuthBackend - LDAP username/password authentication. • GoogleMobileOauth2Backend - Google authentication. • GitHubAuthBackend - GitHub authentication. • ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can set REMOTE_USER in Apache. • DevAuthBackend - Only for development, passwordless login as any user. It’s easy to add more, see the docs on python-social-auth below. The setup documentation for most of these is simple enough that we’ve included it inline in /etc/zulip/settings.py, right above to the settings used to conﬁgure them. The remote user authentication backend is more complex since it requires interfacing with a generic third-party authentication system, and so we’ve documented it in detail below.

The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library. So if you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), you can do this by writing a class similar to GitHubAuthBackend in zproject/backends.py and adding a few settings. Pull requests to add new backends are welcome; they should be tested using the framework in test_auth_backends.py.

Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documents how to conﬁgure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable: (0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend as the only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct set depending on whether your SSO system uses email addresses or just usernames in REMOTE_USER. Make sure that you’ve restarted the Zulip server since making this conﬁguration change.

(1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read: puppet_classes= zulip::voyager, zulip::apache_sso

(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install our SSO integration. (3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example and fill in the configuration required for your SSO service to set REMOTE_USER and place your completed configuration file at /etc/apache2/sites-available/zulip-sso.conf zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication, which is useful for testing quickly. You can set it up by doing the following:

/home/zulip/deployments/current/scripts/restart-server cd/etc/apache2/sites-available/ cp zulip-sso.example zulip-sso.conf htpasswd-c/home/zulip/zpasswd username @example.com # prompts for a password and then continuing with the steps below. (4) Run a2ensite zulip-sso to enable the Apache integration site. (5) Run service apache2 reload to use your new conﬁguration. If Apache isn’t already running, you may need to run service apache2 start instead. Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.

This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not mapping settings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can often help while debugging to temporarily change the Apache config in /etc/apache2/sites-available/zulip-sso to listen on all interfaces rather than just 127.0.0.1 as you debug this. It can also be helpful to change /etc/nginx/zulip-include/app.d/external-sso.conf to proxy_pass to a more explicit URL possibly not over HTTPS when debugging. The following log files can be helpful when debugging this setup: • /var/log/zulip/{errors.log,server.log} (the usual places) • /var/log/nginx/access.log (nginx access logs) • /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to “debug” in the apache config file to make this more verbose) Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summary should help with understanding what’s going on as you try to debug: • Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends.ZulipRemoteUserBackend, zproject/settings.py configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes https://zulip.example.com/ aka the homepage for the main Zulip Django app running behind nginx redirect to /accounts/login/sso if you’re not logged in. • nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888 apache via the config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream localhost:8888 defined in /etc/nginx/zulip-include/upstreams). • The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the htpasswd dialogue; you provide correct login information and the request reaches a second Zulip Django app instance that is running behind Apache with with REMOTE_USER set. That request is served by

zerver.views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in (sets a cookie) or registers the new user (depending whether they have an account). • After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app sees the cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally via username/password). Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the conﬁguration. Sugges- tions for how to improve this SSO setup documentation are very welcome!

This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting the REMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (see http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for documentation on how to set this up and deploy the certificates) to make the DATABASES configuration in zproject/settings.py work (or override that configuration). If you want to use a remote Postgresql database, you should configure the information about the connection with the server. You need a user called “zulip” in your database server. You can configure these options in /etc/zulip/settings.py: • REMOTE_POSTGRES_HOST: Name or IP address of the remote host • REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are: – disable: I don’t care about security, and I don’t want to pay the overhead of encryption. – allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it. – prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it. – require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want. – verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust. – verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it’s the one I specify. Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf: postgres_password= xxxx

Finally, you can stop your database on the Zulip server via: sudo service postgresql stop sudo update-rc.d postgresql disable

In future versions of this feature, we’d like to implement and document how to the remote postgres database server itself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature; if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.

When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:

˓→procpid; which shows the currently running backends and their activity. This is similar to the pg_top output, with the added advantage of showing the complete query, which can be valuable in debugging. To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECT pg_terminate_backend(pid int) as the ‘postgres’ user. The former cancels the backend’s current query and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM to the processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause the database to kill all current connections, roll back any pending transactions, and enter recovery mode.

To start or stop postgres manually, use the pg_ctlcluster command: pg_ctlcluster 9.1[--force] main {start|stop|restart|reload}

By default, using stop uses “smart” mode, which waits for all clients to disconnect before shutting down the database. This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the “fast” mode for shutting down. “Fast” mode is described by the manpage thusly: With the –force option the “fast” mode is used which rolls back all active transactions, disconnects clients immediately and thus shuts down cleanly. If that does not work, shutdown is attempted again in “immediate” mode, which can leave the cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, the postmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failure conditions. This mode should only be used when the machine is about to be shut down. Many database parameters can be adjusted while the database is running. Just modify /etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.

12.4 Debugging issues starting postgres pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check the logs, but you won’t ﬁnd any information there. pg_ctlcluster runs the following command underneath when it actually goes to start Postgres:

Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However, you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since it does bypass some of the work that pg_ctlcluster does.

The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the autovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUM operation, which can often lead to sudden downtime until the operation ﬁnishes. If it did not do this and the age of the oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios alert, perform a VACUUM in each indicated database as a database superuser (postgres). See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for more details on postgres vacuuming.

Zulip offers a wide range of options for how to install the development environment: • Detailed tutorial for Vagrant development environment. Recommended for ﬁrst-time contributors. • Brief installation instructions for Vagrant development environment • Installing on Ubuntu 14.04 Trusty or 16.04 Xenial directly (convenient but more work to maintain/uninstall). • Installing manually on other UNIX platforms • Using Docker (experimental) • Using the Development Environment • Testing If you have a slow network connection, you should probably avoid installing Vagrant (which is large) and either install directly or use the manual install process instead.

This section guides ﬁrst-time contributors through installing the Zulip dev environment on Windows 10, OS X El Capitan, Ubuntu 14.04, and Ubuntu 16.04. The recommended method for installing the Zulip dev environment is to use Vagrant with VirtualBox on Windows and OS X, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and OS X) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Contents: • Requirements • Step 1: Install Prerequisites • Step 2: Get Zulip code • Step 3: Start the dev environment • Step 4: Developing • Troubleshooting & Common Errors If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors. If that doesn’t help, please visit the provision stream in the Zulip developers’ chat for realtime help, or send a note to the Zulip-devel Google group or ﬁle an issue.

Installing the Zulip dev environment requires downloading several hundred megabytes of dependencies. You will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxy to access the internet.) • All: 2GB available RAM, Active broadband internet connection. • OS X: OS X (El Capitan recommended, untested on previous versions), Git, VirtualBox, Vagrant. • Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc. • Windows: Windows 64-bit (Win 10 recommended; Win 7 untested), hardware virtualization enabled (VT-X or AMD-V), administrator access, Cygwin, VirtualBox, Vagrant. Don’t see your system listed above? Check out: • Brief installation instructions for Vagrant development environment • Installing manually on UNIX-based platforms

1. Install VirtualBox 2. Install Vagrant Now you are ready for Step 2: Get Zulip Code.

The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same. If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code: sudo apt-get purge vagrant wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb sudo dpkg-i vagrant *.deb sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir vagrant plugin install vagrant-lxc vagrant lxc sudoers

For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the ofﬁcial Ubuntu repositories. First uninstall any vagrant package you may have installed from the Ubuntu repository: christie@ubuntu-desktop:~ $ sudo apt-get purge vagrant

Now download and install the most recent .deb package from Vagrant: christie@ubuntu-desktop:~ $ wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb christie@ubuntu-desktop:~ $ sudo dpkg -i vagrant*.deb

62 Chapter 14. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

Now install git and lxc-related packages: christie@ubuntu-desktop:~ $ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir

3. Install the vagrant lxc plugin: christie@ubuntu-desktop:~ $ vagrant plugin install vagrant-lxc Installing the 'vagrant-lxc' plugin. This can take a few minutes... Installed the plugin 'vagrant-lxc (1.2.1)'!

If you encounter an error when trying to install the vagrant-lxc plugin, see this.

Finally, conﬁgure sudo to be passwordless when using Vagrant LXC: christie@ubuntu-desktop:~ $ vagrant lxc sudoers [sudo] password for christie:

1. Install Cygwin. Make sure to install default required packages along with git, curl, openssh, and rsync binaries. 2. Install VirtualBox 3. Install Vagrant

In order for symlinks to work within the Ubuntu virtual machine, you must tell Cygwin to create them as native Win- dows symlinks. The easiest way to do this is to add a line to ~/.bash_profile setting the CYGWIN environment variable. Open a Cygwin window and do this: christie@win10 ~ $ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile

Next, close that Cygwin window and open another. If you echo $CYGWIN you should see: christie@win10 ~ $ echo $CYGWIN winsymlinks:native

If you haven’t already created an ssh key and added it to your Github account, you should do that now by following these instructions. 1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged in to Github to do this. 2. Open Terminal (OS X/Ubuntu) or Cygwin (Windows; must run as an Administrator) 3. In Terminal/Cygwin, clone your fork: git clone [email protected]:YOURUSERNAME/zulip.git

This will create a ‘zulip’ directory and download the Zulip code into it. Don’t forget to replace YOURUSERNAME with your git username. You will see something like: christie@win10 ~ $ git clone [email protected]:YOURUSERNAME/zulip.git Cloning into 'zulip'... remote: Counting objects: 73571, done. remote: Compressing objects: 100% (2/2), done. remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569 Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done. Resolving deltas: 100% (51448/51448), done. Checking connectivity... done. Checking out files: 100% (1912/1912), done.`

Change into the zulip directory and tell vagrant to start the Zulip dev environment with vagrant up. christie@win10 ~ $ cd zulip christie@win10 ~/zulip $ vagrant up

The ﬁrst time you run this command it will take some time because vagrant does the following: • downloads the base Ubuntu 14.04 virtual machine image (for OS X and Windows) or container (for Ubuntu) • conﬁgures this virtual machine/container for use with Zulip, • creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at /srv/zulip • runs the tools/provision.py script inside the virtual machine/container, which downloads all required dependencies, sets up the python environment for the Zulip dev environment, and initializes a default test database. You will need an active internet connection during the entire processes. (See Specifying a proxy if you need a proxy to access the internet.) And if you’re running into any problems, please come chat with us in the provision stream of our developers’ chat. Once vagrant up has completed, connect to the dev environment with vagrant ssh:

64 Chapter 14. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

System load: 0.61 Processes: 88 Usage of/: 3.5% of 39.34GB Users logged in:0 Memory usage:7% IP address for eth0: 10.0.2.15 Swap usage:0%

Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud

Congrats, you’re now inside the Zulip dev environment! You can conﬁrm this by looking at the command prompt, which starts with (zulip-venv). Next, start the Zulip server:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ /srv/zulip/tools/run-dev.py --interface=''

As you can see above the application’s root directory, where you can execute Django’s command line utilities is:

2016-05-04 22:20:33,895 INFO: process_fts_updates starting Recompiling templates 2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates done Validating Django models.py... System check identified no issues (0 silenced).

Django version 1.8 Tornado server is running at http://localhost:9993/ Quit the server with CTRL-C. 2016-05-04 18:20:40,716 INFO Tornado loaded0 event queues in 0.001s 2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds Performing system checks...

And ending with something similar to: http://localhost:9994/webpack-dev-server/ webpack result is served from http://localhost:9991/webpack/ content is served from /srv/zulip webpack: bundle is now VALID. 2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds 2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds

Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in your browser on your main machine. You should see something like (this screenshot of the Zulip dev environment). The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your browser, check your terminal and you should see something like:

˓→(unauth via ?) [04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0 2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via

˓→?) [04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0 2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /

You’ll work by editing ﬁles on your host machine, in the directory where you cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++, etc.). When you save changes they will be synced automatically to the Zulip dev environment on the virtual machine/container. Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how this works are available below. Don’t forget to read through the code style guidelines for details about how to conﬁgure your editor for Zulip. For example, indentation should be set to 4 spaces rather than tabs.

It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. See Logging for further details on the run-dev.py console output.

66 Chapter 14. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (OS X/Ubuntu) or Cygwin (Windows) in the directory where you cloned Zulip on your main machine. If you’re new to working with Git/Github, check out this guide.

If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or running tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’ve recently merged changes to the development environment provisioning process that you need to apply to your development environmnet. To update your environment, you’ll need to re-provision your vagrant machine using vagrant provision (or just python tools/provision.py from /srv/zulip inside the Vagrant guest); this should be pretty fast and we’re working to make it faster. See also the documentation on the testing page for how to destroy and rebuild your database if you want to clear out test data.

If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made to the provisioning process, or because you think something is broken), you can do so using vagrant destroy and then vagrant up. This will usually be much faster than the original vagrant up since the base image is already cached on your machine (it takes about 5 minutes to run with a fast Internet connection).

To shut down but preserve the dev environment so you can use it again later use vagrant halt or vagrant suspend. You can do this from the same Terminal/Cygwin window that is running run-dev.py by pressing ^C to halt the server and then typing exit. Or you can halt vagrant from another Terminal/Cygwin window. From the window where run-dev.py is running:

˓→via ?) ^C KeyboardInterrupt (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ exit logout Connection to 127.0.0.1 closed. christie@win10 ~/zulip

christie@win10 ~/zulip $ vagrant suspend ==> default: Saving VM state and suspending execution...

christie@win10 ~/zulip $ vagrant halt ==> default: Attempting graceful shutdown of VM...

When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machine with vagrant ssh and re-start the Zulip server:

christie@win10 ~/zulip $ vagrant up $ vagrant ssh /srv/zulip/tools/run-dev.py --interface=''

At this point you should [read about using the development environment][using-dev-environment.html].

Zulip’s vagrant provisioning process logs useful debugging output to /var/log/zulip_provision.log; if you encounter a new issue, please attach a copy of that ﬁle to your bug report.

The box 'ubuntu/trusty64' could not be found or could not be accessed in the remote catalog. If this is a private box on HashiCorp's Atlas, please verify you're logged in via `vagrant login`. Also, please double-check the name. The expanded URL and error message are shown below: URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]

Then the version of curl that ships with Vagrant is not working on your machine. The ﬁx is simple: replace it with the version from Cygwin. First, determine the location of Cygwin’s curl with which curl:

christie@win10 ~/zulip $ which vagrant /cygdrive/c/HashiCorp/Vagrant/bin/vagrant

68 Chapter 14. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

The path up until /bin/vagrant is what you need to know. In the example above it’s /cygdrive/c/HashiCorp/Vagrant. Finally, copy Cygwin’s curl to Vagrant embedded/bin directory: christie@win10 ~/zulip $ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/

Now re-run vagrant up and vagrant should be able to fetch the required box ﬁle.

==> default: Traceback (most recent call last): ==> default: File"./emoji_dump.py", line 75, in ==> default: ==> default: os.symlink('unicode/{}.png'.format(code_point),'out/ {}.png'.

˓→format(name)) ==> default: OSError ==> default: : ==> default: [Errno 71] Protocol error

Then Vagrant was not able to create a symbolic link. First, if you are using Windows, make sure you have run Cygwin as an administrator. By default, only administrators can create symbolic links on Windows. Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symbolic links for VirtualBox shared folder. You can check to see that this is enabled for your virtual machine with vboxmanage command. Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settings for this virtual machine with vboxmanage getextradata YOURVMNAME enumerate: christie@win10 ~/zulip $ vboxmanage list vms "zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d} christie@win10 ~/zulip $ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1 Key: supported, Value: false

If you see “command not found” when you try to run VBoxManage, you need to add the VirtualBox directory to your path. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\. If vboxmanage enumerate prints nothing, or shows a value of 0 for VBoxInter- nal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command in Terminal/Cygwin: vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/

If you see the following error after running vagrant up: default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying...

A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V). If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system. For further information about troubleshooting vagrant timeout errors see this post.

The tools/provision.py script may encounter an error related to npm install that looks something like:

==> default:+ npm install ==> default: Traceback (most recent call last): ==> default: File"/srv/zulip/tools/provision.py", line 195, in ==> default: ==> default: sys.exit(main()) ==> default: File"/srv/zulip/tools/provision.py", line 191, in main ==> default: ==> default: run(["npm","install"]) ==> default: File"/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run ==> default: ==> default: raise subprocess.CalledProcessError(rc, args) ==> default: subprocess ==> default:. ==> default: CalledProcessError ==> default: : ==> default: Command'['npm','install']' returned non-zero exit status 34 The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong.

Usually this error is not fatal. Try connecting to the dev environment and re-trying the command from withing the virtual machine: christie@win10 ~/zulip $ vagrant ssh (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd /srv/zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip $ npm install npm WARN optional Skipping failed optional dependency /chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.

70 Chapter 14. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_

˓→by' for nil:NilClass (NoMethodError) from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_

˓→plugin' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin' from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run' from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run' from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in

˓→execute' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in èach' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in èxecute' from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in èxecute' from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in èxecute' from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli' from /usr/bin/vagrant:173:in `

And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the issue, which should be ﬁxed when Vagrant 1.8.2 is released. In the meantime, read this post for how to create and apply the patch. It will look something like this: christie@xenial:~ $ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch patching file bundler.rb

14.6.6 Permissions errors when running the test suite in LXC

See “Possible testing issues”.

14.6. Troubleshooting & Common Errors 71 Zulip Documentation, Release 1.4.0

72 Chapter 14. Vagrant environment setup tutorial CHAPTER 15

Vagrant environment setup (in brief)

Start by cloning this repository: git clone https://github.com/zulip/zulip.git This is the recommended approach for all platforms, and will install the Zulip development environment inside a VM or container and works on any platform that supports Vagrant. The best performing way to run the Zulip development environment is using an LXC container on a Linux host, but we support other platforms such as Mac via Virtualbox (but everything will be 2-3x slower). • If your host is Ubuntu 15.04 or newer, you can install and conﬁgure the LXC Vagrant provider directly using apt:

sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir vagrant plugin install vagrant-lxc

You may want to conﬁgure sudo to be passwordless when using Vagrant LXC. • If your host is Ubuntu 14.04, you will need to download a newer version of Vagrant, and then do the following:

sudo apt-get install lxc lxc-templates cgroup-lite redir sudo dpkg-i vagrant *.deb # in directory where you downloaded vagrant vagrant plugin install vagrant-lxc

You may want to configure sudo to be passwordless when using Vagrant LXC. • For other Linux hosts with a kernel above 3.12, follow the Vagrant LXC installation instructions to get Vagrant with LXC for your platform. • If your host is OS X or older Linux, download VirtualBox, download Vagrant, and install them both. • If you’re on OS X and have VMWare, it should be possible to patch Vagrantfile to use the VMWare vagrant provider which should perform much better than Virtualbox. Patches to do this by default if VMWare is available are welcome! • On Windows: You can use Vagrant and Virtualbox/VMWare on Windows with Cygwin, similar to the Mac setup. Be sure to create your git clone using git clone https://github.com/zulip/zulip.git -c core.autocrlf=false to avoid Windows line endings being added to files (this causes weird errors). Once that’s done, simply change to your zulip directory and run vagrant up in your terminal to install the development server. This will take a long time on the first run because Vagrant needs to download the Ubuntu Trusty base image, but later you can run vagrant destroy and then vagrant up again to rebuild the environment and it will be much faster. Once that finishes, you can run the development server as follows:

73 Zulip Documentation, Release 1.4.0

vagrant ssh # Now inside the container /srv/zulip/tools/run-dev.py--interface=''

To get shell access to the virtual machine running the server to run lint, management commands, etc., use vagrant ssh. (A small note on tools/run-dev.py: the --interface='' option will make the development server listen on all network interfaces. While this is correct for the Vagrant guest sitting behind a NAT, you probably don’t want to use that option when using run-dev.py in other environments). At this point you should read about using the development environment.

15.1 Specifying a proxy

If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before running Vagrant up. First, install the Vagrant plugin vagrant-proxyconf: vagrant plugin install vagrant-proxyconf.

Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it for your proxy):

HTTP_PROXY http://proxy_host:port HTTPS_PROXY http://proxy_host:port NO_PROXY localhost,127.0.0.1,.example.com

Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed, you’ll need to run vagrant destroy ﬁrst to clean up the failed installation. You can also change the port on the host machine that Vagrant uses by adding to your ~/.zulip-vagrant-config ﬁle. E.g. if you set:

HOST_PORT 9971

(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development server.

74 Chapter 15. Vagrant environment setup (in brief) CHAPTER 16

Installing directly on Ubuntu

Start by cloning this repository: git clone https://github.com/zulip/zulip.git If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial, you can do that by just running:

# From a clone of zulip.git ./tools/provision.py source/srv/zulip-venv/bin/activate ./tools/run-dev.py # starts the development server

Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrant destroy to clean up the development environment). Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment, ignoring the parts about vagrant (since you’re not using it).

75 Zulip Documentation, Release 1.4.0

76 Chapter 16. Installing directly on Ubuntu CHAPTER 17

Installing manually on UNIX

• Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental) • OpenBSD 5.8 (experimental) • Fedora/CentOS common steps • Steps for all systems If you really want to install everything manually, the below instructions should work. Install the following non-Python dependencies: • libfﬁ-dev — needed for some Python extensions • postgresql 9.1 or later — our database (client, server, headers) • nodejs 0.10 (and npm) • memcached (and headers) • rabbitmq-server • libldap2-dev • python-dev • redis-server — rate limiting • tsearch-extras — better text search • libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs

17.1 On Debian or Ubuntu systems:

17.1.1 Using the ofﬁcial Ubuntu repositories and tsearch-extras deb package:

Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \

77 Zulip Documentation, Release 1.4.0

puppet gettext postgresql

# Next, install Zulip's tsearch-extras postgresql extension # If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras: cd zulip sudo apt-add-repository-yus ppa:tabbott/zulip # On 14.04 sudo apt-get install postgresql-9.3-tsearch-extras # On 16.04 sudo apt-get install postgresql-9.5-tsearch-extras

# Otherwise, you can download a .deb directly # If on 12.04 or wheezy: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-

˓→extras_0.1.2_amd64.deb sudo dpkg-i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb

# If on 14.04: https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-tsearch-

˓→extras_0.1.3_amd64.deb sudo dpkg-i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb

# If on 15.04 or jessie: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-

˓→extras_0.1_amd64.deb sudo dpkg-i postgresql-9.4-tsearch-extras_0.1_amd64.deb

# If on 16.04 or stretch wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-

˓→tsearch-extras_0.2_amd64.deb sudo dpkg-i postgresql-9.5-tsearch-extras_0.2_amd64.deb

Alternatively, you can always build the package from tsearch-extras git. Now continue with the All Systems instructions below.

17.1.2 Using the ofﬁcial Zulip PPA (for 14.04 Trusty or 16.04 Xenial):

Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo add-apt-repository ppa:tabbott/zulip sudo apt-get update sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \ puppet gettext tsearch-extras

Now continue with the All Systems instructions below.

17.2 On Fedora 22 (experimental):

These instructions are experimental and may have bugs; patches welcome!

78 Chapter 17. Installing manually on UNIX Zulip Documentation, Release 1.4.0

Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo dnf install libffi-devel memcached rabbitmq-server \ openldap-devel python-devel redis postgresql-server \ postgresql-devel postgresql libmemcached-devel freetype-devel \ nodejs npm yuicompressor closure-compiler gettext

Now continue with the Common to Fedora/CentOS instructions below.

17.3 On CentOS 7 Core (experimental):

These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Add user zulip to the system (not necessary if you configured zulip # as the administrator user during the install process of CentOS 7). useradd zulip

# Create a password for zulip user passwd zulip

# Allow zulip to sudo visudo # Add this line after line `root ALL=(ALL) ALL` zulip ALL=(ALL) ALL

# Switch to zulip user su zulip

# Enable EPEL 7 repo so we can install rabbitmq-server, redis and # other dependencies sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.

˓→rpm

# Install dependencies sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \ python-devel redis postgresql-server postgresql-devel postgresql \ libmemcached-devel wget python-pip openssl-devel freetype-devel \ libjpeg-turbo-devel zlib-devel nodejs yuicompressor \ closure-compiler gettext

# We need these packages to compile tsearch-extras sudo yum groupinstall"Development Tools"

# clone Zulip's git repo and cd into it cd&& git clone https://github.com/zulip/zulip&& cd zulip/

## NEEDS TESTING: The next few DB setup items may not be required at all. # Initialize the postgres db sudo postgresql-setup initdb

# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf

# Change these lines: host all all 127.0.0.1/32 ident

17.3. On CentOS 7 Core (experimental): 79 Zulip Documentation, Release 1.4.0

host all all::1/128 ident # to this: host all all 127.0.0.1/32 md5 host all all::1/128 md5

Now continue with the Common to Fedora/CentOS instructions below.

17.4 On OpenBSD 5.8 (experimental):

These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \ memcached node libmemcached py-Pillow py-cryptography py-cffi

# Get tsearch_extras and build it (using a modified version which # aliases int4 on OpenBSD): git clone https://github.com/blablacio/tsearch_extras cd tsearch_extras gmake&& sudo gmake install

# Point environment to custom include locations and use newer GCC # (needed for Node modules): export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl" export CXX=eg++

# Create tsearch_data directory: sudo mkdir/usr/local/share/postgresql/tsearch_data

# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/local/share/postgresql/tsearch_data/english.stop sudo touch/usr/local/share/postgresql/tsearch_data/en_us.dict sudo touch/usr/local/share/postgresql/tsearch_data/en_us.affix

Finally continue with the All Systems instructions below.

17.5 Common to Fedora/CentOS instructions

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Build and install postgres tsearch-extras module wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.

˓→tar.gz tar xvzf tsearch-extras_0.1.3.tar.gz cd ts2 make sudo make install

# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us

80 Chapter 17. Installing manually on UNIX Zulip Documentation, Release 1.4.0

# package. sudo touch/usr/share/pgsql/tsearch_data/english.stop sudo touch/usr/share/pgsql/tsearch_data/en_us.dict sudo touch/usr/share/pgsql/tsearch_data/en_us.affix

# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf

# Add this line before the first uncommented line to enable password # auth: host all all 127.0.0.1/32 md5

# Start the services sudo systemctl start redis memcached rabbitmq-server postgresql

# Enable automatic service startup after the system startup sudo systemctl enable redis rabbitmq-server memcached postgresql

Finally continue with the All Systems instructions below.

17.6 All Systems:

Make sure you have followed the steps speciﬁc for your platform: • Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental) • OpenBSD 5.8 (experimental) • Fedora/CentOS For managing Zulip’s python dependencies, we recommend using virtualenvs. You must create two virtualenvs. One for Python 2 and one for Python 3. You must also install appropriate python packages in them. You should either install the virtualenvs in /srv, or put symlinks to them in /srv. If you don’t do that, some scripts might not work correctly. You can run tools/setup/setup_venvs.py to do this. This script will create two virtualenvs - /srv/zulip-venv and /srv/zulip-py3-venv. If you want to do it manually, here are the steps: virtualenv/srv/zulip-venv-p python2 # Create a python2 virtualenv source/srv/zulip-venv/bin/activate # Activate python2 virtualenv pip install--upgrade pip # upgrade pip itself because older versions have known

˓→issues pip install--no-deps-r requirements/py2_dev.txt # install python packages required

˓→for development virtualenv/srv/zulip-py3-venv-p python3 # Create a python3 virtualenv source/srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv pip install--upgrade pip # upgrade pip itself because older versions have known

˓→issues pip install--no-deps-r requirements/py3_dev.txt # install python packages required

˓→for development

17.6. All Systems: 81 Zulip Documentation, Release 1.4.0

Now run these commands:

./tools/setup/install-phantomjs ./tools/install-mypy ./tools/setup/download-zxcvbn ./tools/setup/emoji_dump/build_emoji ./scripts/setup/generate_secrets.py -d if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_

˓→english.stop /var/postgresql/tsearch_data/; else sudo cp ./puppet/zulip/files/ ˓→postgresql/zulip_english.stop /usr/share/postgresql/9.*/tsearch_data/; fi ./scripts/setup/configure-rabbitmq ./tools/setup/postgres-init-dev-db ./tools/do-destroy-rebuild-database ./tools/setup/postgres-init-test-db ./tools/do-destroy-rebuild-test-database ./manage.py compilemessages npm install

If npm install fails, the issue may be that you need a newer version of npm. You can use npm install -g npm to update your version of npm and try again. To start the development server:

./tools/run-dev.py

. . . and visit http://localhost:9991/.

17.6.1 Proxy setup for by-hand installation

If you are building the development environment on a network where a proxy is required to access the Internet, you will need to set the proxy in the environment as follows: • On Ubuntu, set the proxy environment variables using: export https_proxy=http://proxy_host:port export http_proxy=http://proxy_host:port

• And set the npm proxy and https-proxy using: npm config set proxy http://proxy_host:port npm config set https-proxy http://proxy_host:port

82 Chapter 17. Installing manually on UNIX CHAPTER 18

Using Docker (experimental)

Start by cloning this repository: git clone https://github.com/zulip/zulip.git The docker instructions for development are experimental, so they may have bugs. If you try them and run into any issues, please report them! You can also use Docker to run a Zulip development environment. First, you need to install Docker in your development machine following the instructions. Some other interesting links for somebody new in Docker are: • Get Started • Understand the architecture • Docker run reference • Dockerﬁle reference Then you should create the Docker image based on Ubuntu Linux, ﬁrst go to the directory with the Zulip source code: docker build-t user/zulipdev.

Now you’re going to install Zulip dependencies in the image: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash $ /usr/bin/python /srv/zulip/tools/provision.py --docker docker ps -af ancestor=user/zulipdev docker commit -m "Zulip installed" user/zulipdev:v2

Finally you can run the docker server with: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \ /srv/zulip/tools/start-dockers

If you want to connect to the Docker instance to build a release tarball you can use: docker ps docker exec -it /bin/bash $ source /home/zulip/.bash_profile $ $ exit

To stop the server use: docker ps docker kill

83 Zulip Documentation, Release 1.4.0

If you want to run all the tests you need to start the servers ﬁrst, you can do it with: docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash $ tools/test-all-docker

You can modify the source code in your development machine and review the results in your browser.

84 Chapter 18. Using Docker (experimental) CHAPTER 19

Using the Development Environment

Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, the development server homepage just shows a list of the users that exist on the server and you can login as any of them by just clicking on a user. This setup saves time for the common case where you want to test something other than the login process; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION case of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d like to test. While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulip development server encounters. When you make a change, here’s a guide for what you need to do in order to see your change take effect in Develop- ment: • If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’ll just need to reload the browser window to see changes take effect. The Handlebars frontend HTML templates (static/templates) are automatically recompiled by the tools/compile-handlebars-templates job, which runs as part of tools/run-dev.py. • If you change Python code used by the the main Django/Tornado server processes, these services are run on top of Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado servers whenever you save changes to Python code. You can watch this happen in the run-dev.py console to make sure the backend has reloaded. • The Python queue workers will also automatically restart when you save changes. However, you may need to ctrl-C and then restart run-dev.py manually if a queue worker has crashed. • If you change the database schema, you’ll need to use the standard Django migrations process to create and then run your migrations; see the new feature tutorial for an example. Additionally you should check out the detailed testing docs for how to run the tests properly after doing a migration. (In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will take care of running migrations and then cleanly restaring the server for you).

85 Zulip Documentation, Release 1.4.0

86 Chapter 19. Using the Development Environment CHAPTER 20

Writing a new integration

Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to making integrating with Zulip and getting you integration merged upstream so everyone else can beneﬁt from it as easy as possible while maintaining the high quality of the Zulip integrations library. On this page you’ll ﬁnd: • An overvew of the different types of integrations possible with Zulip. • General advice for writing integrations. • Details about writing webhook integrations. • Details about writing Python script and plugin integrations. • A guide to documenting your integration. •A detailed walkthrough of a simple “Hello World” integration. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help writing integration, please email [email protected], open an issue, or submit a pull request to share your ideas!

20.1 Types of integrations

We have several different ways that we integrate with 3rd part products, ordered here by which types we prefer to write: 1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting content to a particular URI on our site with data about the event. For these, you usually just need to add a new handler in zerver/views/webhooks.py (plus test/document/etc.). An example commit implementing a new webhook is: https://github.com/zulip/zulip/pull/324. 2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (by shelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrations in our API release tarballs (by writing the integration in api/integrations). 3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existing software. These are often more work, but for some products are the only way to integrate with the product at all.

87 Zulip Documentation, Release 1.4.0

20.2 General advice

• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g. emoji, markdown emphasis, @-mentions, or !avatar(email)). • Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes for much better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for all messages; for an integration like Nagios, put the service in the topic. • Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to providing options for triggering Zulip messages only for certain message types, certain projects, or sending different messages to different streams/topics, to make it easy for teams to configure the integration to support their workflow. • Consistently capitalize the name of the integration in the documentation and the Client name the way the vendor does. It’s OK to use all-lower-case in the implementation. • Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use – sometimes the right API is just not properly documented. • A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via your local Zulip dev environment. This enables you to do end-to-end testing with live data from the service you’re integrating and can help you spot why something isn’t working or if the service is using custom HTTP headers.

20.3 Webhook integrations

New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use the right process. Here’s how we recommend doing it: • First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’re integrating. You can use these captured payloads to create a set of test fixtures for your integration under zerver/fixtures. • Then write a draft webhook handler under zerver/views/webhooks/; there are a lot of examples in that directory. We recommend templating off a short one (like stash.py or zendesk.py), since the longer ones usually just have more complex parsing which can obscure what’s common to all webhook integrations. In addition to writing the integration itself, you’ll need to create Integration object and add it to WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py'; search forwebhook‘ in that file to find the existing ones (and please add yours in the alphabetically correct place). • Then write a test for your fixture in zerver/tests/test_hooks.py, and you can iterate on the tests and webhooks handler until they work, all without ever needing to post directly from the server you’re integrating to your Zulip development machine. To run just the tests from the test class you wrote, you can use e.g.

test-backend zerver.tests.test_hooks.PagerDutyHookTests

See this guide for more details on the Zulip test runner. • Once you’ve gotten your webhook working and passing a test, capture payloads for the other common types of posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast. Webhook integration tests should all use ﬁxtures (as opposed to contacting the service), since otherwise the tests can’t run without Internet access and some sort of credentials for the service. • Finally, write documentation for the integration; there’s a detailed guide below. See the Hello World webhook Walkthrough below for a detailed look at how to write a simple webhook.

88 Chapter 20. Writing a new integration Zulip Documentation, Release 1.4.0

20.3.1 Files that need to be created

Select a name for your webhook and use it consistently. The examples below are for a webhook named ‘MyWebHook’. • static/images/integrations/logos/mywebhook.png: An image to represent your integration in the user interface. Generally this Should be the logo of the platform/server/product you are integrating. See Documenting your integration for details. • static/images/integrations/mywebbook/001.png: A screen capture of your integration for use in the user interface. You can add as many images as needed to effectively document your webhook integration. See Documenting your integration for details. • zerver/fixtures/mywebhook/mywebhook_messagetype.json: Sample json payload data used by tests. Add one ﬁxture ﬁle per type of message supported by your integration. See Testing and writing tests for details. • zerver/views/webhooks/mywebhook.py: Includes the main webhook integration function including any needed helper functions.

20.3.2 Files that need to be updated

• templates/zerver/integrations.html: Edit to add end-user documentation. See Documenting your integration for details. • zerver/test_hooks.py: Edit to include tests for your webbook. See Testing and writing tests for details. • zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONS to register it. This will automatically register a url for the webhook of the form api/v1/external/mywebhook and associate with the function called api_mywebhook_webhook in zerver/views/webhooks/mywebhook.py.

20.4 Python script and plugin integrations

For plugin integrations, usually you will need to consult the documentation for the third party software in order to learn how to write the integration. But we have a few notes on how to do these: • You should always send messages by POSTing to URLs of the form https://zulip.example.com/v1/messages/, not the legacy /api/v1/send_message message sending API. • We usually build Python script integration with (at least) 2 files: ‘zulip_foo_config.py‘‘ containing the configuration for the integration including the bots’ API keys, plus a script that reads from this configuration to actually do the work (that way, it’s possible to update the script without breaking users’ configurations). • Be sure to test your integration carefully and document how to install it (see notes on documentation below). • You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identify the integration and version number, separated by a slash. If possible, you should collect platform information and include that in ()s after the version number. Some examples of ideal UAs are:

ZulipDesktop/0.7.0 (Ubuntu; 14.04) ZulipJenkins/0.1.0 (Windows; 7.2) ZulipMobile/0.5.4 (Android; 4.2; maguro)

20.4. Python script and plugin integrations 89 Zulip Documentation, Release 1.4.0

20.5 Documenting your integration

Every Zulip integration must be documented in templates/zerver/integrations.html. Usually, this in- volves a few steps: • Make sure you’ve added your integration to zerver/lib/integrations.py; this results in your integration appearing on the /integrations page. You’ll need to add a logo image for your integration under the static/images/integrations/logos/.png, where is the name of the integration, all in lower case. • Add an integration-instructions class block also in the alphabetically correct place, explaining all the steps required to setup the integration, including what URLs to use, etc. If there are any screens in the product involved, take a few screenshots with the input fields filled out with sample values in order to make the instructions really easy to follow. For the screenshots, use something like [email protected] for the email addresses and an obviously fake API key like abcdef123456790. • Finally, generate a message sent by the integration and take a screenshot of the message to provide an example message in the documentation. If your new integration is a webhook integration, you can generate such a message from your test fixtures using send_webhook_fixture_message:

./manage.py send_webhook_fixture_message \ --fixture=zerver/fixtures/pingdom/pingdom_imap_down_to_up.json \ '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'

When generating the screenshot of a sample message, give your test bot a nice name like “GitHub Bot”, use the project’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not just the message body. When writing documentation for your integration, be sure to use the {{ external_api_uri }} template variable, so that your integration documentation will provide the correct URL for whatever server it is deployed on. If special conﬁguration is required to set the SITE variable, you should document that too, inside an {% if api_site_required %} check.

20.6 Hello World webhook Walkthrough

Below explains each part of a simple webhook integration, called Hello World. This webhook sends a “hello” message to the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receives in the http request. Use this walkthrough to learn how to write your ﬁrst webhook integration.

20.6.1 Step 0: Create ﬁxtures

The ﬁrst step in creating a webhook is to examine the data that the service you want to integrate will be sending to Zulip. You can use requestb.in or a similar tool to capture webook payload(s) from the service you are integrating. Examining this data allows you to do two things: 1. Determine how you will need to structure your webook code, including what message types your integration should support and how; and, 2. Create ﬁxtures for your webook tests.

90 Chapter 20. Writing a new integration Zulip Documentation, Release 1.4.0

Fixtures enable the testing of webhook integration code without the need to actually contact the service being integrated. Because Hello World is a very simple webhook that does one thing, it requires only one ﬁxture, zerver/fixtures/helloworld/helloworld_hello.json:

{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe", }

When writing your own webhook integration, you’ll want to write a test function for each distinct message condition your webhook supports. You’ll also need a corresponding ﬁxture for each of these tests. See Step 3: Create tests or Testing for further details.

20.6.2 Step 1: Create main webhook code

The majority of the code for your webhook integration will be in a single python file in zerver/views/webhooks/. The name of this file should be the name of your webhook, all lower-case, with file extension .py: mywebhook.py. The Hello World integration is in zerver/views/webhooks/helloworld.py: from __future__ import absolute_import from django.utils.translation import ugettext as _ from zerver.lib.actions import check_send_message from zerver.lib.response import json_success, json_error from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view from zerver.lib.validator import check_dict, check_string from zerver.models import Client, UserProfile from django.http import HttpRequest, HttpResponse from six import text_type from typing import Dict, Any, Iterable, Optional

@api_key_only_webhook_view('HelloWorld') @has_request_variables def api_helloworld_webhook(request, user_profile, client, payload=REQ(argument_type='body'), stream=REQ(default='test'), topic=REQ(default='Hello World')): # type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]],

˓→text_type, Optional[text_type]) -> HttpResponse

# construct the body of the message body='Hello! I am happy to be here! :smile:'

# try to add the Wikipedia article of the day # return appropriate error if not successful try: body_template=' \nThe Wikipedia featured article for today is **[{featured_ ˓→title}]({featured_url})**' body+= body_template.format( **payload) except KeyError as e: return json_error(_("Missing key {} in JSON").format(str(e)))

20.6. Hello World webhook Walkthrough 91 Zulip Documentation, Release 1.4.0

# send the message check_send_message(user_profile, client,'stream', [stream], topic, body)

# return json result return json_success()

The above code imports the required functions and defines the main webhook function api_helloworld_webook, decorating it with api_key_only_webhook_view and has_request_variables. You must pass the name of your webhook to the api_key_only_webhook_view decorator. Here we have used HelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating in camel case, spelled as the product spells its own name (except always first letter upper-case). You should name your webhook function as such api_webhookname_webhook where webhookname is the name of your webhook and is always lower-case. At minimum, the webhook function must accept request (Django HttpRequest object), user_profile (Zulip’s user object), and client (Zulip’s analogue of UserAgent). You may also want to define additional parameters using the REQ object. In the example above, we have defined payload which is populated from the body of the http request, stream with a default of test (available by default in Zulip dev environment), and topic with a default of Hello World. The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate your webhook functions. In the body of the function we define the body of the message as Hello! I am happy to be here! :smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as provided by the json payload. If the json payload does not include data for featured_title and featured_url we catch a KeyError and use json_error to return the appropriate information: a 400 http status code with relevant details. Then we send a public (stream) message with check_send_message which will validate the message and then send it. Finally, we return a 200 http status with a JSON format success message via json_success().

20.6.3 Step 2: Create an api endpoint for the webhook

In order for a webhook to be externally available, it must be mapped to a url. This is done in zerver/lib/integrations.py. Look for the lines beginning with:

WEBHOOK_INTEGRATIONS=[

And you’ll ﬁnd the entry for Hello World:

WebhookIntegration('helloworld', display_name='Hello World'),

This tells the Zulip api to call the api_helloworld_webhook function in zerver/views/webhooks/helloworld.py when it receives a request at /api/v1/external/helloworld. This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page using static/images/integrations/logos/helloworld.png as its icon. At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough code to test your integration.

92 Chapter 20. Writing a new integration Zulip Documentation, Release 1.4.0

You can do so by using Zulip itself or curl on the command line. Using manage.py from within Zulip Dev environment:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/fixtures/helloworld/helloworld_hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key='

After which you should see something similar to:

2016-07-07 15:06:59,187 INFO 127.0.0.1 POST 200 143ms (mem:6ms/13) (md:

˓→43ms/1) (db: 20ms/9q) (+start: 147ms)/api/v1/external/helloworld (helloworld-

˓→[email protected] via ZulipHelloWorldWebhook)

Using curl: curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn

˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://

˓→localhost:9991/api/v1/external/helloworld\?api_key\=

After which you should see:

{"msg":"","result":"success"}

Using either method will create a message in Zulip:

20.6.4 Step 3: Create tests

Every webhook integraton should have a corresponding test class in zerver/tests/test_hooks.py. You should name the class HookTests and this class should accept WebhookTestCase. For our HelloWorld webhook, we name the test class HelloWorldHookTests: class HelloWorldHookTests(WebhookTestCase): STREAM_NAME='test' URL_TEMPLATE="/api/v1/external/helloworld?&api_key= {api_key}" FIXTURE_DIR_NAME='helloworld'

# Note: Include a test function per each distinct message condition your

˓→integration supports def test_hello_message(self): # type: () -> None expected_subject= u"Hello World"; expected_message= u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/ ˓→Marilyn_Monroe)**";

# use fixture named helloworld_hello self.send_and_test_stream_message('hello', expected_subject, expected_message, content_type="application/x-www-form-

˓→urlencoded")

20.6. Hello World webhook Walkthrough 93 Zulip Documentation, Release 1.4.0

def get_body(self, fixture_name): # type: (text_type) -> text_type return self.fixture_data("helloworld", fixture_name, file_type="json")

When writing tests for your webook, you’ll want to include one test function (and corresponding ﬁxture) per each distinct message condition that your integration supports. If, for example, we added support for sending a goodbye message to our Hello World webook, we would add another test function to HelloWorldHookTests class called something like test_goodbye_message:

def test_goodbye_message(self): # type: () -> None expected_subject= u"Hello World"; expected_message= u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";

# use fixture named helloworld_goodbye self.send_and_test_stream_message('goodbye', expected_subject, expected_

˓→message, content_type="application/x-www-form-

˓→urlencoded")

As well as a new ﬁxture helloworld_goodbye.json in zerver/fixtures/helloworld/:

{ "featured_title":"Goodbye", "featured_url":"https://en.wikipedia.org/wiki/Goodbye", }

Once you have written some tests, you can run just these new tests from within the Zulip dev environment with this command:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend zerver.tests.test_hooks.HelloWorldHookTests

(Note: You must run the tests from /srv/zulip directory.) You will see some script output and if all the tests have passed, you will see:

Running zerver.tests.test_hooks.HelloWorldHookTests.test_hello_message DONE!

20.6.5 Step 4: Create documentation

Next, we add end-user documentation for our webhook integration to templates/zerver/integrations.html. There are two parts to the end-user documentation on this page. The ﬁrst is a div with class integration-lozenge for each integration. This div shows the logo of your webhook, its name, and a link to its installation and usage instructions. Because there is an entry for the Hello World webhook in WEBHOOK_INTEGRATIONS in zerver/lib/integratins.py, this div will be generated automatically. The second part is a div with the webhook’s usage instructions:

94 Chapter 20. Writing a new integration Zulip Documentation, Release 1.4.0

Learn how Zulip integrations work with this simple Hello World example!

The Hello World webhook will use the test stream, which is created by default in the Zulip dev environment. If you are running Zulip in production, you should make sure this stream exists.

Next, on your Zulip settings page, create a Hello World bot. Construct the URL for the Hello World bot using the API key and stream name: {{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&

˓→stream=test

To trigger a notication using this webhook, use `send_webhook_fixture_message`

˓→from the Zulip command line:

  (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/fixtures/helloworld/helloworld_hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key='
 
Or, use curl:
  curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":
˓→"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }'
˓→http://localhost:9991/api/v1/external/helloworld\?api_key\=
 
Congratulations! You're done!
 Your messages may look like:

These documentation blocks should fall alphabetically. For the integration-lozenge div this happens automatically when the html is generated. For the integration-instructions div, we have added the div between the blocks for Github and Hubot, respectively. See Documenting your integration for further details, including how to easily create the message screenshot.

20.6.6 Step 5: Preparing a pull request to zulip/zulip

When you have ﬁnished your webhook integration and are ready for it to be available in the Zulip product, follow these steps to prepare your pull request: 1. Run tests including linters and ensure you have addressed any issues they report. See Testing for details. 2. Read through Code styles and conventions and take a look through your code to double-check that you’ve followed Zulip’s guidelines. 3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control for tips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll want to squash your changes into a single commit and include a good, clear commit message. 4. Push code to your fork.

20.6. Hello World webhook Walkthrough 95 Zulip Documentation, Release 1.4.0

5. Submit a pull request to zulip/zulip. If you would like feedback on your integration as you go, feel free to submit pull requests as you go, preﬁxing them with [WIP].

96 Chapter 20. Writing a new integration CHAPTER 21

Writing a new application feature

The changes needed to add a new feature will vary, of course, but this document provides a general outline of what you may need to do, as well as an example of the speciﬁc steps needed to add a new feature: adding a new option to the application that is dynamically synced through the data system in real-time to all browsers the user may have open.

21.1 General Process in brief

21.1.1 Adding a ﬁeld to the database

Update the model: The server accesses the underlying database in zerver/ models.py. Add a new ﬁeld in the appropriate class. Create and run the migration: To create and apply a migration, run:

./manage.py makemigrations ./manage.py migrate

Test your changes: Once you’ve run the migration, restart memcached on your development server (/etc/init.d/memcached restart) and then restart run-dev.py to avoid interacting with cached objects.

21.1.2 Backend changes

Database interaction: Add any necessary code for updating and interacting with the database in zerver/lib/actions.py. It should update the database and send an event announcing the change. Application state: Modify the fetch_initial_state_data and apply_events functions in zerver/lib/actions.py to update the state based on the event you just created. Backend implementation: Make any other modiﬁcations to the backend required for your change. New views: Add any new application views to zerver/urls.py. This includes both views that serve HTML (new pages on Zulip) as well as new API endpoints that serve JSON-formatted data. Testing: At the very least, add a test of your event data ﬂowing through the system in test_events.py.

21.1.3 Frontend changes

JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact ﬁles you may need to change depend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it to static/js/server_events.js.

97 Zulip Documentation, Release 1.4.0

CSS: The primary CSS ﬁle is static/styles/zulip.css. If your new feature requires UI changes, you may need to add additional CSS to this ﬁle. Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. For JavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part of the build/deploy process. Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackbox tests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. The unit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information on writing and running tests see the testing documentation.

21.2 Example Feature

This example describes the process of adding a new setting to Zulip: a ﬂag that restricts inviting new users to admins only (the default behavior is that any user can invite other users). It is based on an actual Zulip feature, and you can review the original commit in the Zulip git repo. (Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the migration format has changed.)

21.2.1 Update the model

First, update the database and model to store the new setting. Add a new boolean ﬁeld, invite_by_admins_only, to the Realm model in zerver/models.py.

--- a/zerver/models.py +++ b/zerver/models.py @@ -139,6 +139,7 @@ class Realm(ModelReprMixin, models.Model): restricted_to_domain = models.BooleanField(default=True) # type: bool invite_required = models.BooleanField(default=False) # type: bool + invite_by_admins_only = models.BooleanField(default=False) # type: bool create_stream_by_admins_only = models.BooleanField(default=False) # type: bool mandatory_topics = models.BooleanField(default=False) # type: bool

21.2.2 Create the migration

Create the migration ﬁle: ./manage.py makemigrations. Make sure to commit the generated ﬁle to git: git add zerver/migrations/NNNN_realm_invite_by_admins_only.py (NNNN is a number that is equal to the number of migrations.) If you run into problems, the Django migration documentation is helpful.

21.2.3 Test your migration changes

Apply the migration: ./manage.py migrate Output: shell $ ./manage.py migrate Operations to perform: Synchronize unmigrated apps: staticfiles, analytics, pipeline Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,

˓→auth, contenttypes Synchronizing apps without migrations:

98 Chapter 21. Writing a new application feature Zulip Documentation, Release 1.4.0

Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying zerver.0026_realm_invite_by_admins_only... OK

21.2.4 Handle database interactions

Next, we will move on to implementing the backend part of this feature. Like typical apps, we will need our backend to update the database and send some response to the client that made the request. Beyond that, we need to orchestrate notifications to other clients (or other users, if you will) that our setting has changed. Clients find out about settings through two closely related code paths. When a client first contacts the server, the server sends the client its initial state. Subsequently, clients subscribe to “events,” which can (among other things) indicate that settings have changed. For the backend piece, we will need our action to make a call to send_event to send the event to clients that are active. We will also need to modify fetch_initial_state_data so that future clients see the new changes. Anyway, getting back to implementation details... In zerver/lib/actions.py, create a new function named do_set_realm_invite_by_admins_only. This function will update the database and trigger an event to notify clients when this setting changes. In this case there was an existing realm|update event type which was used for setting similar flags on the Realm model, so it was possible to add a new property to that event rather than creating a new one. The property name matches the database field to make it easy to understand what it indicates. The second argument to send_event is the list of users whose browser sessions should be notified. Depending on the setting, this can be a single user (if the setting is a personal one, like time display format), only members in a particular stream or all active users in a realm. :

# zerver/lib/actions.py

def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only): realm.invite_by_admins_only= invite_by_admins_only realm.save(update_fields=['invite_by_admins_only']) event= dict( type="realm", op="update", property='invite_by_admins_only', value=invite_by_admins_only, ) send_event(event, active_user_ids(realm)) return {}

21.2.5 Update application state

You then need to add code that will handle the event and update the application state. In zerver/lib/actions.py update the fetch_initial_state and apply_events functions. :

def fetch_initial_state_data(user_profile, event_types, queue_id): # ... state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`

21.2. Example Feature 99 Zulip Documentation, Release 1.4.0

In this case you don’t need to change apply_events because there is already code that will correctly handle the realm update event type: :

def apply_events(state, events, user_profile): for event in events: # ... elif event['type'] =='realm': field='realm_'+ event['property'] state[field]= event['value']

21.2.6 Add a new view

You then need to add a view for clients to access that will call the newly-added actions.py code to update the database. This example feature adds a new parameter that should be sent to clients when the application loads and be accessible via JavaScript, and there is already a view that does this for related ﬂags: update_realm. So in this case, we can add out code to the existing view instead of creating a new one. :

# zerver/views/__init__.py

def home(request): # ... page_params= dict( # ... realm_invite_by_admins_only= register_ret['realm_invite_by_admins_only'], # ... )

Since this feature also adds a checkbox to the admin page, and adds a new property the Realm model that can be modiﬁed from there, you also need to make changes to the update_realm function in the same ﬁle: :

# zerver/views/__init__.py

def update_realm(request, user_profile, name=REQ(validator=check_string, default=None), restricted_to_domain=REQ(validator=check_bool, default=None), invite_by_admins_only=REQ(validator=check_bool,default=None)):

# ...

if invite_by_admins_only is not None and realm.invite_by_admins_only != invite_by_admins_only: do_set_realm_invite_by_admins_only(realm, invite_by_admins_only) data['invite_by_admins_only']= invite_by_admins_only

Then make the required front end changes: in this case a checkbox needs to be added to the admin page (and its value added to the data sent back to server when a realm is updated) and the change event needs to be handled on the client. To add the checkbox to the admin page, modify the relevant template, static/templates/admin_tab.handlebars (omitted here since it is relatively straightforward). Then add code to handle changes to the new form control in static/js/admin.js.:

var url = "/json/realm"; var new_invite_by_admins_only = $("#id_realm_invite_by_admins_only").prop("checked"); data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);

100 Chapter 21. Writing a new application feature Zulip Documentation, Release 1.4.0

channel.patch({ url: url, data: data, success: function (data) { # ... if (data.invite_by_admins_only) { ui.report_success("New users must be invited by an admin!", invite_by_admins_

˓→only_status); } else { ui.report_success("Any user may now invite new users!", invite_by_admins_only_

˓→status); } # ... } });

Finally, update server_events.js to handle related events coming from the server. :

# static/js/server_events.js function dispatch_normal_event(event) { switch (event.type) { # ... case'realm': if (event.op ==='update'&& event.property ==='invite_by_admins_only'){ page_params.realm_invite_by_admins_only= event.value; } } }

Any code needed to update the UI should be placed in dispatch_normal_event callback (rather than the channel.patch) function. This ensures the appropriate code will run even if the changes are made in another browser window. In this example most of the changes are on the backend, so no UI updates are required.

21.2. Example Feature 101 Zulip Documentation, Release 1.4.0

102 Chapter 21. Writing a new application feature CHAPTER 22

Writing views in Zulip

22.1 What this covers

This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide, and treat this as a reference. If you have experience with Django, much of this will be familiar, but you may want to read about how REST requests are dispatched, and how request authentication works. This document supplements the new feature tutorial and the testing documentation.

22.2 What is a view?

A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’t show a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations, which shows the integration documentation. These paths show up in the address bar of the browser. There are other views that are only seen by software, namely the API views. They are used to build the various clients that Zulip has, namely the web client (which is also used by the desktop client) and the mobile clients.

22.3 Modifying urls.py

A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serve HTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations). The format of the URL patterns in Django is documented here, and the Zulip speciﬁc details for these are discussed in detail in the life of a request doc. We have two Zulip-speciﬁc conventions we use for internationalization and for our REST API, respectively.

22.4 Writing human-readable views

If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py

i18n_urls = [ ... + url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html

˓→')),

103 Zulip Documentation, Release 1.4.0

+ url(r'^postcards/$', 'zerver.views.postcards'), ]

As an example, if a request comes in for Spanish, language code es, the server path will be something like: es/features/.

22.4.1 Decorators used for webpage views

This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators. require_post:

@require_post def accounts_register(request): # type: (HttpRequest) -> HttpResponse

This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page is requested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll get a HTTP 405 METHOD NOT ALLOWED error. zulip_login_required: This decorator veriﬁes that the browser is logged in (i.e. has a valid session cookie) before providing the view for this route, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If a request comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork of Django’s login_required, adding a few extra checks speciﬁc to Zulip.

@zulip_login_required def home(request): # type: (HttpRequest) -> HttpResponse

22.4.2 Writing a template

Templates for the main website are found in templates/zerver.

22.5 Writing API REST endpoints

These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-string responses. Almost all Zulip view code is in the implementations of API REST endpoints. The REST API does authentication of the user through rest_dispatch, which is documented in detail at zerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser, or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.

>>> import requests >>> r= requests.get('https://api.github.com/user', auth=('[email protected]',

˓→'0123456789abcdeFGHIJKLmnopQRSTUV')) >>> r.status_code -> 200

104 Chapter 22. Writing views in Zulip Zulip Documentation, Release 1.4.0

22.5.1 Request variables

Most API views will have some arguments that are passed as part of the request to control the behavior of the view. In any well-engineered view, you need to write code to parse and validate that the arguments exist and have the correct form. For many applications, this leads to one of serveral bad outcomes: • The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users of the API. • Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error messages. • Every view function comes with another function that does the validation that has the problems from the last bullet point. In Zulip, we solve this problem with a the special decorator called has_request_variables which allows a developer to declare the arguments a view function takes and validate their types all within the def line of the function. We like this framework because we have found it makes the validation code compact, readable, and conveniently located in the same place as the method it is validating arguments for. Here’s an example:

from zerver.decorator import has_request_variables, REQ, JsonableError, \ require_realm_admin

@require_realm_admin @has_request_variables def create_user_backend(request, user_profile, email=REQ(), password=REQ(), full_name=REQ(), short_name=REQ()): # ... code here

You will notice the special REQ() in the keyword arguments to create_user_backend. has_request_variables parses the declared keyword arguments of the decorated function, and for each that has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request, parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the event that an argument is missing, doesn’t parse as JSON, or otherwise is invalid. require_realm_admin is another decorator which checks the authorization of the given user_profile to make sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily to show how has_request_variables should be the inner decorator. The implementation of has_request_variables is documented in detail in zerver/lib/request.py) REQ also helps us with request variable validation. For example: • msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP parameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keyword argument. • streams_raw = REQ("subscriptions",validator=check_list(check_string)) will check that the “subscriptions” HTTP parameter is a list of strings, marshalled as JSON, and pass it into the function with the Python keyword argument streams_raw. • message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTP parameter is a string containing a non-negative integer (converter differs from validator in that it does not automatically marshall the input from JSON). See zerver/lib/validator.py for more validators and their documentation.

22.5. Writing API REST endpoints 105 Zulip Documentation, Release 1.4.0

22.5.2 Deciding which HTTP verb to use

When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read or write operation. If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a resource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and used by clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as it happens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more. If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’re creating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If you want to create the data multiple times for multiple requests (for example, requesting the send_message view multiple times with the same content should send multiple messages), you should use POST. If you’re updating existing data, use PATCH. If you’re removing data, use DELETE.

22.5.3 Idempotency

When writing a new API endpoint, with the exception of things like sending messages, requests should be safe to repeat, without impacting the state of the server. This is idempotency. You will often want to return an error if a request to change something would do nothing because the state is already as desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be the same, but the repeated requests won’t change the server more than once or cause unwanted side effects.

22.5.4 Making changes to the database

If the view does any modiﬁcation to the database, that change is done in a helper function in zerver/lib/actions.py. Those functions are responsible for doing a complete update to the state of the server, which often entails both updating the database and sending any events to notify clients about the state change. When possible, we prefer to design a clean boundary between the view function and the actions function is such that all user input validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for atomically executing the change (this is usually signalled by having the actions function have a name starting with do_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection to the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a name starting with check_. For example, in zerver/views/init.py:

@require_realm_admin @has_request_variables def update_realm(request, user_profile, name=REQ(validator=check_string,

˓→default=None),...)): # type: (HttpRequest, UserProfile, ...) -> HttpResponse realm= user_profile.realm data={} # type: Dict[str, Any] if name is not None and realm.name != name: do_set_realm_name(realm, name) data['name']='updated' and in zerver/lib/actions.py:

106 Chapter 22. Writing views in Zulip Zulip Documentation, Release 1.4.0

def do_set_realm_name(realm, name): # type: (Realm, text_type) -> None realm.name= name realm.save(update_fields=['name']) event= dict( type="realm", op="update", property='name', value=name, ) send_event(event, active_user_ids(realm)) realm.save() actually saves the changes to the realm to the database, and send_event sends the event to active clients belonging to the provided list of users (in this case, all altive users in the Zulip realm).

22.5.5 Calling from the web application

You should always use channel. to make an HTTP call to the Zulip JSON API. As an example, in static/js/admin.js var url= "/json/realm"; var data={ name: JSON.stringify(new_name), } channel.patch({ url: url, data: data, success: function (response_data) { if (response_data.name !== undefined){ ui.report_success(i18n.t("Name changed!"), name_status); } ...

22.5.6 Calling from an API client

Here’s how you might manually make a call from python: payload={'name': new_name}

# email and API key api_auth=('[email protected]','0123456789abcdeFGHIJKLmnopQRSTUV') r= requests.patch(SERVER_URL+'api/v1/realm', data=json.dumps(payload), auth=api_auth, )

This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a nice interface for accessing the API.

22.5. Writing API REST endpoints 107 Zulip Documentation, Release 1.4.0

22.6 Legacy endpoints used by the web client

New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage of a browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your feature available to other clients, especially the mobile clients. These endpoints make use of some older authentication decorators, authenticated_json_api_view, authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.

22.7 Webhook integration endpoints

Webhooks are called by other services, often to send a message as part of those services’ integrations. They are most often POST requests, and often there is very little you can customize about them. Usually you can expect that the webhook for a service will allow specification for the target server for the webhook, and an API key. If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator, to fill in the user_profile and client fields of a request:

@api_key_only_webhook_view('PagerDuty') @has_request_variables def api_pagerduty_webhook(request, user_profile, client, payload=REQ(argument_type='body'), stream=REQ(default='pagerduty'), topic=REQ(default=None)):

The client will be the result of get_client("ZulipPagerDutyWebhook") in this example.

108 Chapter 22. Writing views in Zulip CHAPTER 23

Life of a Request

It can sometimes be confusing to ﬁgure out how to write a new feature, or debug an existing one. Let us try to follow a request through the Zulip codebase, and dive deep into how each part works. We will use as our example the creation of users through the API, but we will also highlight how alternative requests are handled.

23.1 A request is sent to the server, and handled by Nginx

When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know how this works, except for when it isn’t working. Nginx does the first level of routing–deciding which application will serve the request (or deciding to serve the request itself for static content). In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static, and are served unminified.

23.2 Nginx secures trafﬁc with SSL

If you visit your Zulip server in your browser and discover that your trafﬁc isn’t being properly encrypted, an nginx misconﬁguration is the likely culprit.

23.3 Static ﬁles are served directly by Nginx

Static ﬁles include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3). location/static/{ alias/home/zulip/prod-static/; error_page 404/static/html/404.html; }

23.4 Nginx routes other requests between tornado and django

All our connected clients hold open long-polling connections so that they can recieve events (messages, presence notiﬁcations, and so on) in real-time. Events are served by Zulip’s tornado application.

109 Zulip Documentation, Release 1.4.0

Nearly every other kind of request is served by the zerver Django application. Here is the relevant nginx routing conﬁguration.

23.5 Django routes the request to a view in urls.py ﬁles

There are various urls.py ﬁles throughout the server codebase, which are covered in more detail in the directory structure doc. The main Zulip Django app is zerver. The routes are found in

zproject/urls.py zproject/legacy_urls.py

There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types of requests are handled, and focus on how the REST API handles our user creation example.

23.6 Views serving HTML are internationalized by server path

If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of the browser, and serve HTML. For example, the /hello page (preview here) gets translated in Chinese at zh-cn/hello/ (preview here). Note the zh-cn preﬁx–that url pattern gets added by i18n_patterns.

23.7 API endpoints use REST

Our example is a REST API endpoint. It’s a PUT to /users. With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-out endpoints, Zulip uses REST for its API. This means that we use: • POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb is appropriate. • PUT for creating something for which we have a unique ID. • DELETE for deleting something • PATCH for updating or editing attributes of something. • GET to get something (read-only) • HEAD to check the existence of something to GET, without getting it; useful to check a link without downloading a potentially large link • OPTIONS (handled automatically, see more below) Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the request multiple times and get the same state on the server. You might get a different response after the ﬁrst request, as we like to give our clients an error so they know that no new change was made by the extra requests. POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH is special–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible. This cookbook and tutorial can be helpful if you are new to REST web applications.

110 Chapter 23. Life of a Request Zulip Documentation, Release 1.4.0

23.7.1 PUT is only for creating new things

If you’re used to using PUT to update or modify resources, you might find our convention a little strange. We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (like sending a message with the same content multiple times), and PATCH to modify resources. In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.

23.7.2 OPTIONS

The OPTIONS method will yield the allowed methods. This request: OPTIONS https://zulip.tabbott.net/api/v1/users yields a response with this HTTP header: Allow: PUT,GET We can see this reﬂected in zproject/urls.py:

url(r'^users$','zerver.lib.rest.rest_dispatch', {'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'}),

In this way, the API is partially self-documenting.

23.7.3 Legacy endpoints are used by the web client

The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the web client, and use POST. You can see them in zproject/legacy_urls.py.

23.7.4 Webhook integrations may not be RESTful

Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. They are likely to use only POST.

23.8 Django calls rest_dispatch for REST endpoints, and authenticates

For requests that correspond to a REST url pattern, Zulip conﬁgures its url patterns (see zerver/lib/rest.py) so that the action called is rest_dispatch. This method will authenticate the user, either through a session token from a cookie, or from an email:api-key string given via HTTP Basic Auth for API clients. It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then ﬁgure out which view to show from that. In our example,

{'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'}

is supplied as an argument to rest_dispatch, along with the HTTPRequest. The request has the HTTP verb PUT, which rest_dispatch can use to ﬁnd the correct view to show: zerver.views.users.create_user_backend.

23.8. Django calls rest_dispatch for REST endpoints, and authenticates 111 Zulip Documentation, Release 1.4.0

23.9 The view will authorize the user, extract request variables, and validate them

This is covered in good detail in the writing views doc

23.10 Results are given as JSON

Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of an error, which gives a JSON string: {'result': 'error','msg': } in a HTTP Response with a content type of ‘application/json’. To pass back data from the server to the calling client, in the event of a successfully handled request, we use json_success(data=. This will result in a JSON string: {'result': 'success','msg': '','data'='{'var_name1': 'var_value1','var_name2': 'var_value2'...} with a HTTP 200 status and a content type of ‘application/json’. That’s it!

112 Chapter 23. Life of a Request CHAPTER 24

Version control

24.1 Commit Discipline

We follow the Git project’s own commit discipline practice of “Each commit is a minimal coherent idea”. This discipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commit history a much more useful resource for developers trying to understand why the code works the way it does, which also helps a lot in preventing bugs. Coherency requirements for any commit: • It should pass tests (so test updates needed by a change should be in the same commit as the original change, not a separate “fix the tests that were broken by the last commit” commit). • It should be safe to deploy individually, or comment in detail in the commit message as to why it isn’t (maybe with a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checks in a future commit should be avoided – the security checks should be there from the beginning. • Error handling should generally be included along with the code that might trigger the error. • TODO comments should be in the commit that introduces the issue or functionality with further work required. When you should be minimal: • Significant refactorings should be done in a separate commit from functional changes. • Moving code from one file to another should be done in a separate commits from functional changes or even refactoring within a file. • 2 different refactorings should be done in different commits. • 2 different features should be done in different commits. • If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did, you probably should have just done 2 commits. When not to be overly minimal: • For completely new features, you don’t necessarily need to split out new commits for each little subfeature of the new feature. E.g. if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty of options/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob of new code isn’t fun, so please be thoughtful about submitting things in reviewable units. • Don’t bother to split back end commits from front end commits, even though the backend can often be coherent on its own. Other considerations:

113 Zulip Documentation, Release 1.4.0

• Overly fine commits are easily squashed, but not vice versa, so err toward small commits, and the code reviewer can advise on squashing. • If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, not writing a new “fix tests” commit on top of it. Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them; it’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix your history using git rebase -i. It can take some practice to get used to writing your commits with a clean history so that you don’t spend much time doing interactive rebases. For example, often you’ll start adding a feature, and discover you need to a refactoring part- way through writing the feature. When that happens, we recommend stashing your partial feature, do the refactoring, commit it, and then finish implementing your feature.

24.2 Commit Messages

• The ﬁrst line of commit messages should be written in the imperative and be kept relatively short while concisely explaining what the commit does. For example: Bad: bugfix gather_subscriptions was broken fix bug #234.

Good:

Fix gather_subscriptions throwing an exception when given bad input.

• Use present-tense action verbs in your commit messages. Bad:

Fixing gather_subscriptions throwing an exception when given bad input. Fixed gather_subscriptions throwing an exception when given bad input.

Good:

Fix gather_subscriptions throwing an exception when given bad input.

• Please use a complete sentence in the summary, ending with a period. • The rest of the commit message should be written in full prose and explain why and how the change was made. If the commit makes performance improvements, you should generally include some rough benchmarks showing that it actually improves the performance. • When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue is automatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph of the commit message read e.g. “Fixes: #123.” • Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so that your commit message will be reasonably readable in git log in a normal terminal. • In your commit message, you should describe any manual testing you did in addition to running the automated tests, and any aspects of the commit that you think are questionable and you’d like special attention applied to.

114 Chapter 24. Version control CHAPTER 25

Code style and conventions

25.1 Be consistent!

Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other code has actively bad style, ﬁx it (in a separate commit). When in doubt, send an email to [email protected] with your question.

25.2 Lint tools

You can run them all at once with

./tools/lint-all

You can set this up as a local Git commit hook with

``tools/setup-git-repo``

The Vagrant setup process runs this for you. lint-all runs many lint checks in parallel, including • JavaScript (JSLint) tools/jslint/check-all.js contains a pretty fine-grained set of JSLint options, rule exceptions, and allowed global variables. If you add a new global, you’ll need to add it to the list. • Python (Pyflakes) • templates • Puppet configuration • custom checks (e.g. trailing whitespace and spaces-not-tabs)

25.3 Secrets

Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function in zproject/settings.py to read secrets from /etc/zulip/secrets.conf.

115 Zulip Documentation, Release 1.4.0

25.4 Dangerous constructs

25.4.1 Misuse of database queries

Look out for Django code like this:

[Foo.objects.get(id=bar.x.id) for bar in Bar.objects.filter(...) if bar.baz<7]

This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of a list comprehension, write a single query using Django’s QuerySet API. If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t defer this optimization when performing schema changes, or else you may later ﬁnd that it’s impossible.

25.4.2 UserProﬁle.objects.get() / Client.objects.get / etc.

In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Instead always use get_user_profile_by_{email,id}. There are 3 reasons for this: 1. It’s guaranteed to correctly do a case-inexact lookup 2. It fetches the user object from remote cache, which is faster 3. It always fetches a UserProﬁle object which has been queried using .selected_related(), and thus will perform well when one later accesses related models like the Realm. Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remote cache.

25.4.3 Using Django model objects as keys in sets/dicts

Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing with objects obtained from different database queries: For example, UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)]) is False You should work with the IDs instead.

25.4.4 user_proﬁle.save()

You should always pass the update_fields keyword argument to .save() when modifying an existing Django model object. By default, .save() will overwrite every value in the column, which results in lots of race conditions where unrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfile object before the first thread wrote out its change.

25.4.5 Using raw saves to update important model objects

In most cases, we already have a function in zephyr/lib/actions.py with a name like do_activate_user that will correctly handle lookups, caching, and notifying running browsers via the event system about your change. So please check whether such a function exists before writing new code to modify a model object, since your new code has a good chance of getting at least one of these things wrong.

116 Chapter 25. Code style and conventions Zulip Documentation, Release 1.4.0

25.4.6 x.attr(’zid’) vs. rows.id(x)

Our message row DOM elements have a custom attribute zid which contains the numerical message ID. Don’t access this directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrong result, occasionally, just enough to make a bug that’s impossible to track down. You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Even in cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts in JavaScript where a string is needed, you can pass a number without any explicit conversion.

25.4.7 JavaScript var

Always declare JavaScript variables using var: var x=...;

In a function, var is necessary or else x will be a global variable. For variables declared at global scope, this has no effect, but we do it for consistency. JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts the same as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variables at the top of a function.

25.4.8 JavaScript for (i in myArray)

Don’t use it: [1], [2], [3]

25.4.9 jQuery global state

Don’t mess with jQuery global state once the app has loaded. Code like this is very dangerous:

$.ajaxSetup({ async: false }); $.get(...); $.ajaxSetup({ async: true }); jQuery and the browser are free to run other code while the request is pending, which could perform other Ajax requests with the altered settings. Instead, switch to the more general $.ajax_ function, which can take options like async.

25.4.10 State and logs ﬁles

Do not write state and logs ﬁles inside the current working directory in the production environment. This will not how you expect, because the current working directory for the app changes every time we do a deploy. Instead, hardcode a path in settings.py – see SERVER_LOG_PATH in settings.py for an example.

25.5 JS array/object manipulation

For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We used to use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer more choices.

25.5. JS array/object manipulation 117 Zulip Documentation, Release 1.4.0

A quick conversion table:

$.each → _.each (parameters to the callback reversed) $.inArray → _.indexOf (parameters reversed) $.grep → _.filter $.map → _.map $.extend → _.extend

There’s a subtle difference in the case of _.extend; it will replace attributes with undeﬁned, whereas jQuery won’t:

$.extend({foo: 2}, {foo: undefined}); // yields {foo: 2}, BUT... _.extend({foo: 2}, {foo: undefined}); // yields {foo: undefined}!

Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. If you’re doing this, you probably want _.find, _.every, or _.any, rather than ‘each’. Some Underscore functions have multiple names. You should always use the canonical name (given in large print in the Underscore documentation), with the exception of _.any, which we prefer over the less clear ‘some’.

25.6 More arbitrary style things

25.6.1 General

Indentation is four space characters for Python, JS, CSS, and shell scripts. Indentation is two space characters for HTML templates. We never use tabs anywhere in source code we write, but we have some third-party files which contain tabs. Keep third-party static files under the directory zephyr/static/third/, with one subdirectory per third-party project. We don’t have an absolute hard limit on line length, but we should avoid extremely long lines. A general guideline is: refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to 120 or so. Whitespace guidelines: • Put one space (or more for alignment) around binary arithmetic and equality operators. • Put one space around each part of the ternary operator. • Put one space between keywords like if and while and their associated open paren. • Put one space between the closing paren for if and while-like constructs and the opening curly brace. Put the curly brace on the same line unless doing otherwise improves readability. • Put no space before or after the open paren for function calls and no space before the close paren for function calls. • For the comma operator and colon operator in languages where it is used for inline dictionaries, put no space before it and at least one space after. Only use more than one space for alignment.

25.6.2 JavaScript

Don’t use == and != because these operators perform type coercions, which can mask bugs. Always use === and !==. End every statement with a semicolon.

118 Chapter 25. Code style and conventions Zulip Documentation, Release 1.4.0

if statements with no braces are allowed, if the body is simple and its extent is abundantly clear from context and formatting. Anonymous functions should have spaces before and after the argument list:

var x= function (foo, bar) {//...

When calling a function with an anonymous function as an argument, use this style:

$.get('foo', function (data) { var x = ...; // ... });

The inner function body is indented one level from the outer function call. The closing brace for the inner function and the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate for calls with multiple anonymous functions or other arguments following them. Use

$(function () { ...

rather than

$(document).ready(function () { ...

and combine adjacent on-ready functions, if they are logically related. The best way to build complicated DOM elements is a Mustache template like zephyr/static/templates/message.handlebars. For simpler things you can use jQuery DOM building APIs like so:

var new_tr = $('').attr('id', zephyr.id);

Passing a HTML string to jQuery is ﬁne for simple hardcoded things:

foo.append('

foo

');
 but avoid programmatically building complicated strings. We used to favor attaching behaviors in templates like so:
 but there are some reasons to prefer attaching events using jQuery code: • Potential huge performance gains by using delegated events where possible • When calling a function from an onclick attribute, this is not bound to the element like you might think • jQuery does event normalization Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.
25.6.3 HTML / CSS
Don’t use the style= attribute. Instead, deﬁne logical classes and put your styles in external ﬁles such as zulip.css.
25.6. More arbitrary style things 119 Zulip Documentation, Release 1.4.0
Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’t have to care if the tag type changes in the future. Don’t use inline event handlers (onclick=, etc. attributes). Instead, attach a jQuery event handler ($('#foo').on('click',function () {...})) when the DOM is ready (inside a $(function () {...}) block). Use this format when you have the same block applying to multiple CSS styles (separate lines for each selector): selector1, selector2 { };
25.6.4 Python
• Scripts should start with #!/usr/bin/env python and not #/usr/bin/python (the right Python may not be installed in /usr/bin) or #/usr/bin/env/python2.7 (bad for Python 3 compatibility). Don’t put a shebang line on a Python ﬁle unless it’s meaningful to run it as a script. (Some libraries can also be run as scripts, e.g. to run a test suite.) • The ﬁrst import in a ﬁle should be from __future__ import absolute_import, per PEP 328 • Put all imports together at the top of the ﬁle, absent a compelling reason to do otherwise. • Unpacking sequences doesn’t require list brackets:
[x, y]= xs # unnecessary x, y= xs # better
• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple. • When selecting by id, don’t use foo.pk when you mean foo.id. E.g.
 recipient= Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
 should be written as
 recipient= Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
 in case we ever change the primary keys.
25.6.5 Tests
All signiﬁcant new features should come with tests. See testing.
25.6.6 Third party code
When adding new third-party packages to our codebase, please include “[third]” at the beginning of the commit message. You don’t necessarily need to do this when patching third-party code that’s already in tree.
120 Chapter 25. Code style and conventions CHAPTER 26
Testing and writing tests
26.1 Overview
Zulip has a full test suite that includes many components. The most important components are documented in depth in their own sections: • Django: backend Python tests • Casper: end-to-end UI tests • Node: unit tests for JS front end code • Linters This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot database issues, how to manually test the front end, and how to plan for the future upgrade to Python3.
26.2 Running tests
Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter the Vagrant environment before running the tests: vagrant ssh cd/srv/zulip
Then, to run the full Zulip test suite, do this:
./tools/test-all
This runs the linter (tools/lint-all) plus all of our test suites; they can all be run separately (just read tools/test-all to see them). You can also run individual tests which can save you a lot of time debugging a test failure, e.g.:
./tools/lint-all # Runs all the linters in parallel ./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube ./tools/test-js-with-casper 09-navigation.js ./tools/test-js-with-node utils.js
The above setup instructions include the ﬁrst-time setup of test databases, but you may need to rebuild the test database occasionally if you’re working on new database migrations. To do this, run:
121 Zulip Documentation, Release 1.4.0
./tools/do-destroy-rebuild-test-database
26.2.1 Possible testing issues
• When running the test suite, if you get an error like this:
 sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_
˓→array(unknown, text, tsquery) does not exist LINE2:...ECT message_id, flags, subject, rendered_content, ts_match_l... ^
. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and the do-destroy-rebuild*-database scripts. • When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.
26.3 Schema and initial data changes
If you change the database schema or change the initial test data, you have to regenerate the pristine test database by running tools/do-destroy-rebuild-test-database.
26.4 Wiping the test databases
You should ﬁrst try running: tools/do-destroy-rebuild-test-database If that fails you should try to do:
 sudo-u postgres psql > DROP DATABASE zulip_test; > DROP DATABASE zulip_test_template;
 and then run tools/do-destroy-rebuild-test-database
26.4.1 Recreating the postgres cluster
 warning This is irreversible, so do it with care, and never do this anywhere in production. If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repair it, you can recreate it. On Ubuntu:
 sudo pg_dropcluster--stop 9.1 main sudo pg_createcluster--locale=en_US.utf8--start 9.1 main
122 Chapter 26. Testing and writing tests Zulip Documentation, Release 1.4.0
26.5 Manual testing (local app + web browser)
26.5.1 Clearing the manual testing database
You can use:
./tools/do-destroy-rebuild-database to drop the database on your development environment and repopulate your it with the Shakespeare characters and some test messages between them. This is run automatically as part of the development environment setup process, but is occasionally useful when you want to return to a clean state for testing.
26.5.2 JavaScript manual testing debug.js has some tools for proﬁling JavaScript code, including: • ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console. • ‘IterationProﬁler‘: Proﬁle part of looping constructs (like a for loop or $.each). You mark sections of the iteration body and the IterationProﬁler will sum the costs of those sections over all iterations. Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. They both have proﬁlers, but Chrome’s is a sampling proﬁler while Firebug’s is an instrumenting proﬁler. Using them both can be helpful because they provide different information.
26.6 Python 3 Compatibility
Zulip is working on supporting Python 3, and all new code in Zulip should be Python 2+3 compatible. We have converted most of the codebase to be compatible with Python 3 using a suite of 2to3 conversion tools and some manual work. In order to avoid regressions in that compatibility as we continue to develop new features in Zulip, we have a special tool, tools/check-py3, which checks all code for Python 3 syntactic compatibility by running a subset of the automated migration tools and checking if they trigger any changes. tools/check-py3 is run automatically in Zulip’s Travis CI tests (in the ‘static-analysis’ build) to avoid any regressions, but is not included in test-all since it is quite slow. To run tools/check-py3, you need to install the modernize and future Python packages (which are included in requirements/py3k.txt, which itself is included in requirements/dev.txt, so you probably already have these packages installed). To run check-py3 on just the Python ﬁles in a particular directory, you can change the current working directory (e.g. cd zerver/) and run check-py3 from there.
26.5. Manual testing (local app + web browser) 123 Zulip Documentation, Release 1.4.0
124 Chapter 26. Testing and writing tests CHAPTER 27
Python static type checker (mypy)
 mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip is using mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written inside comments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip: user_dict={} # type: Dict[str, UserProfile] def get_user_profile_by_email(email): # type: (str) -> UserProfile ... # Actual code of the function here
You can learn more about it at: • The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotations used by mypy correctly. • The Python 2 type annotation syntax spec in PEP 484 • Using mypy with Python 2 code The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the ‘static-analysis’ build.
27.1 type_debug.py zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters of the decorated function and the return type whenever that function is called. This can help ﬁnd out what parameter types a function is supposed to accept, or if parameters with the wrong types are being passed to a function. Here is an example using the interactive console:
>>> from zerver.lib.type_debug import print_types >>> >>> @print_types ... def func(x, y): ... return x+y ... >>> func(1.0,2) func(float, int) -> float 3.0 >>> func('a','b') func(str, str) -> str 'ab' >>> func((1,2), (3,))
125 Zulip Documentation, Release 1.4.0
 func((int, int), (int,)) -> (int, int, int) (1, 2, 3) >>> func([1,2,3], [4,5,6,7]) func([int, ...], [int, ...]) -> [int, ...] [1, 2, 3, 4, 5, 6, 7] print_all prints the type of the ﬁrst item of lists. So [int,...] represents a list whose ﬁrst element’s type is int. Types of all items are not printed because a list can have many elements, which would make the output too large. Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1: 'a',2: 'b',3: 'c'} will be printed as {int: str,...}.
27.2 Zulip goals
Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays that way. Our current coverage is shown in Coveralls.
27.3 Installing mypy
If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3 virtualenv at zulip-py3-venv (mypy only supports Python 3). If it isn’t installed (e.g. because you haven’t reprovisioned recently), you can run tools/install-mypy to install it.
27.4 Running mypy on Zulip’s code locally
To run mypy on Zulip’s python code, run the command: tools/run-mypy
It will output errors in the same style as a compiler would. For example, if your code has a type error like this: foo=1 foo='1' you’ll get an error like this: test.py: note: In function"test": test.py:200: error: Incompatible types in assignment (expression has type"str",
˓→variable has type"int")
If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbott on your pull request (or email [email protected]) to get help; we’d love to both build a great trou- bleshooting guide in this doc and also help contribute improvements to error messages upstream. Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinned version of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.
126 Chapter 27. Python static type checker (mypy) Zulip Documentation, Release 1.4.0
27.5 Excluded ﬁles
Since several Python ﬁles in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list of ﬁles to be excluded from the check for CI is present in tools/run-mypy. To run mypy on all Python ﬁles, ignoring the exclude list, you can pass the --all option to tools/run-mypy. tools/run-mypy--all
If you type annotate some of those ﬁles so that they pass without errors, please remove them from the exclude list.
27.6 Mypy is there to ﬁnd bugs in Zulip before they impact users
For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a wide range of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to get to the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include: • A bug in any new type annotations you added. • A bug in the existing type annotations. • A bug in Zulip! • Some Zulip code is correct but confusingly reuses variables with different types. • A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project). Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way that makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @shar- maeklavya2 or @timabbott on the relevant pull request or issue on GitHub. If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHub repository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.
27.7 Annotating strings
In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to support languages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters like mathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCII characters. That’s why unicode was introduced in Python 2. But there are problems with the unicode and str system. Implicit conversions between str and unicode use the ascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people who always write in English. To minimize such implicit conversions, we should have a strict separation between str and unicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately it doesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking) insist that parameters passed to them are str. To make our code work correctly in Python 2, we have to identify strings which contain data which could come from non-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These strings should be unicode. We also have to identify strings which should be str like Exception names, attribute names, parameter names, etc. Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static type checking will tell us if we are incorrectly mixing the two. However, unicode is not deﬁned in Python 3. We want
27.5. Excluded ﬁles 127 Zulip Documentation, Release 1.4.0 our code to be Python 3 compatible in the future. This can be achieved using ‘six’, a Python 2 and 3 compatibility library. six.text_type is deﬁned as str in Python 3 and as unicode in Python 2. We’ll be using text_type (instead of unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from six import text_type and using text_type for annotation instead of doing import six and using six.text_type for annotation, because text_type is used so extensively for type annotations that we don’t need to be that verbose. Sometimes you’ll ﬁnd that you have to convert strings from one type to another. zerver/lib/str_utils.py has utility functions to help with that. It also has documentation (in docstrings) which explains the right way to use them.
128 Chapter 27. Python static type checker (mypy) CHAPTER 28
Settings system
The page documents the Zulip settings system, and hopefully should help you decide how to correctly implement new settings you’re adding to Zulip. We have two types of administrative settings in Zulip: server settings (which are set via conﬁguration ﬁles are apply to the whole Zulip installation), and realm settings (which are usually set via the /#administration page in the Zulip web application) and apply to a single Zulip realm/organization (which for most Zulip servers is the only realm on the server). Philosophically, the goals of the settings system are to make it convenient for: • Zulip server administrations to conﬁgure Zulip’s featureset for their server without needing to patch Zulip • Realm administrators to conﬁgure settings for their organization independently without needing to talk with the server administrator. • Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable conﬁguration.
28.1 Server settings
Zulip uses the Django settings system, which means that the settings ﬁles are Python programs that set a lot of variables with all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django code using e.g.: from django.conf import settings print(settings.EMAIL_GATEWAY_PATTERN)
Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), you can use e.g.:
$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN %s@localhost:9991
Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the ~1000 lines of settings needed by the Zulip Django app. As a result, there are a few ﬁles involved in the Zulip settings for server administrations. In a production environment, we have: • /etc/zulip/settings.py (the template is in the Zulip repo at zproject/prod_settings_template.py) is the main system administration facing settings ﬁle for Zulip. It contains all the server-speciﬁc settings, such as how to send outgoing email, the hostname of the Postgres database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in zproject/settings.py, and then override it here.
129 Zulip Documentation, Release 1.4.0
• /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate-secrets.py as part of installation) contains secrets used by the Zulip installation. These are read using the standard Python ConfigParser, and accessed in zproject/settings.py by the get_secret function. All secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the get_secret function in zproject/settings.py. • zproject/settings.py is the main Django settings ﬁle for Zulip. It contains all the settings that are constant for all Zulip installations (e.g. conﬁguration for logging, static assets, middleware, etc.), as well as default values for the settings the user would set in /etc/zulip/settings.py (you can look at the DEFAULT_SETTINGS dictionary to easily review the settings available). zproject/settings.py has a line from prod_settings import *, which has the effect of im- porting /etc/zulip/settings.py in a prod environment (via a symlink). In a development environment, we have zproject/settings.py, and additionally: • zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just im- ports prod_settings_template.py. • zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf. • zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend and Casper), which are applied on top of the development environment settings. When adding a new server setting to Zulip, you will typically add it in two or three places: • In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments. If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the user will add the value when they conﬁgure the feature). • In an appropriate section of zproject/prod_settings_template.py, with documentation in the com- ments explaining the settings’s purpose and effect. • Possibly also zproject/dev_settings.py, if the desired value of the setting for Zulip development en- vironments is different from the default for production (and similarly for zproject/test_settings.py). Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’s features; they should be enabled by default in production if we expect most Zulip sites to want those settings.
28.1.1 Testing non-default settings
You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However, this only works for settings which are checked at runtime, not settings which are only accessed in initialization of Django (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.
28.2 Realm settings
Realm settings are preferred for any conﬁguration that is a matter of organizational policy (as opposed to technical capabilities of the server). As a result, conﬁguration options for user-facing functionality is almost always added as a new realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be replaced with realm settings: • NAME_CHANGES_DISABLED • INLINE_IMAGE_PREVIEW • ENABLE_GRAVATAR
130 Chapter 28. Settings system Zulip Documentation, Release 1.4.0
• Which authentication methods are allowed should probably appear in both places; in server settings indicating the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants to allow users to login with.
28.2. Realm settings 131 Zulip Documentation, Release 1.4.0
132 Chapter 28. Settings system CHAPTER 29
Queue processors
Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: • Asynchronously doing expensive operations like sending email notiﬁcations which can take seconds per email and thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm). • Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g. UserActivityInternal) which don’t have any immediate runtime effect. • Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. • Processing mobile push notiﬁcations and email mirroring system messages. • Processing various errors, frontend tracebacks, and slow database queries in a batched fashion. • Doing markdown rendering for messages delivered to the Tornado via websockets. Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s in critical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients. We use the pika library to interface with RabbitMQ, using a simple custom integration deﬁned in zerver/lib/queue.py.
29.1 Adding a new queue processor
To add a new queue processor: • Deﬁne the processor in zerver/worker/queue_processors.py using the @assign_queue decora- tor; it’s pretty easy to get the template for an existing similar queue processor. This sufﬁces to test your queue worker in the Zulip development environment (tools/run-dev.py will automatically restart the queue pro- cessors and start running your new queue processor code). You can also run a single queue processor manually using e.g. ./manage.py process_queue --queue=user_activity. • So that supervisord will known to run the queue processor in production, you will need to deﬁne a program entry for it in servers/puppet/modules/zulip/files/supervisor/conf.d/zulip.conf and add it to the zulip-workers group further down in the ﬁle. • For monitoring, you need to add a check that your worker is running to puppet/zulip/ﬁles/cron.d/rabbitmq- numconsumers if it’s a one-at-a-time consumer like user_activity_internal or a custom nagios check if it is a bulk processor like slow_queries.
133 Zulip Documentation, Release 1.4.0
29.2 Publishing events into a queue
You can publish events to a RabbitMQ queue using the queue_json_publish function deﬁned in zerver/lib/queue.py.
29.3 Clearing a RabbitMQ queue
If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue , for example:
./manage.py purge_queue user_activity
You can also use the amqp tools directly. Install amqp-tools from apt and then run: amqp-delete-queue--username=zulip--password='...'--server=localhost \ --queue=user_presence with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.
134 Chapter 29. Queue processors CHAPTER 30
Unread counts and the pointer
When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you? Conceptually, Zulip takes you to the place where you left off (e.g. the ﬁrst unread message), not the most recent messages, to facilitate reviewing all the discussions that happened while you were away from your computer. The scroll position is then set to keep that message in view and away from both the top and bottom of the visible section of messages. But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currently decides which message to select, along with some notes on improvements we’d like to make to the model. First a bit of terminology: • “Narrowing” is the process of ﬁltering to a particular subset of the messages the user has access to. • The blue cursor box (the “pointer”) is around is called the “selected” message. Zulip ensures that the currently selected message is always in-view.
30.1 Pointer logic
30.1.1 Recipient bar: message you clicked
If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list at the top of a group of messages), Zulip will select the the message you clicked on. This provides a nice user experience where you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly the same scroll position in the window after the narrowing as it was at before.
30.1.2 Search or sidebar click: unread/recent matching narrow
If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulip will instead selected on the ﬁrst unread message matching that narrow, or if there are none, the most recent messages matching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enough messages you’ev seen before still in view at the top to provide you with context), which is usually what you want. (When ﬁnding the “ﬁrst unread message”, Zulip ignores unread messages in muted streams or in muted topics within non-muted streams.)
135 Zulip Documentation, Release 1.4.0
30.1.3 Unnarrow: previous sequence
When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selected in the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped forward to the ﬁrst unread and non-muted message in the home view (or the bottom of the feed if there is none). This makes for a nice experience reading threads via the home view in sequence.
30.1.4 New home view: “high watermark”
When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /), Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic around unnarrowing in the last bullet, this is usually just before the ﬁrst unread message in the home view, but if you never go to the home view, or you leave messages unread on some streams in your home view, this can lag. We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.
30.1.5 Narrow in a new tab: closest to pointer
When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer, which is what you would have got had you loaded the browser window to your home view and then clicked on the nearest message matching your narrow (which might have been offscreen). We plan to change this to match the Search/sidebar behavior.
30.1.6 Forced reload: state preservation
When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when a new version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preserve the state – what (if any) narrow the user was in, the selected message, and even exact scroll position! For more on the user experience philosophy guiding these decisions, see the architectural overview.
30.2 Unread count logic
How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might use the product. The algorithm is as follows: • Any message which is selected or above a message which is selected is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. • If the whitspace at the very bottom of the feed is in view, all messages in view are marked as read. These two simple rules, combined with the pointer logic above, end up matching user expectations well for whether the product should treat them as having read a set of messages (or not).
136 Chapter 30. Unread counts and the pointer CHAPTER 31
Markdown implementation
Zulip has a special ﬂavor of Markdown, currently called ‘bugdown’ after Zulip’s original name of “humbug”. End users are using Bugdown within the client, not original Markdown. Zulip has two implementations of Bugdown. The ﬁrst is based on Python-Markdown (zerver/lib/bugdown/) and is used to authoritatively render messages on the backend (and implements expensive features like querying the Twitter API to render tweets nicely). The other is in JavaScript, based on marked (static/js/echo.js), and is used to preview and locally echo messages the moment the sender hits enter, without waiting for round trip from the server. The two implementations are tested for compatibility via zerver/tests/test_bugdown.py and the ﬁxtures under zerver/fixtures/bugdown-data.json. The JavaScript implementation knows which types of messages it can render correctly, and thus while there is code to rerender messages based on the authoritative backend rendering (which would clause a change in the rendering visible only to the sender shortly after a message is sent), this should never happen, and whenever it does it is considered a bug. Instead, if the frontend doesn’t know how to correctly render a message, we simply won’t echo the message for the sender until it’s rendered by the backend. So for example, a message containing a link to Twitter will not be rendered by the JavaScript implementation because it doesn’t support doing the 3rd party API queries required to render tweets nicely. I should note that the below documentation is based on a comparison with original Markdown, not newer Markdown variants like CommonMark.
31.1 Zulip’s Markdown philosophy
Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis to bug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that is helps more than getting in the way. The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to ﬁx the formatting after typing it the ﬁrst time. While that’s basically ﬁne when writing a blog, it gets annoying very fast in a chat product; even though you can edit messages to ﬁx formatting mistakes, you don’t want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip: • What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it to render in a reasonable way? This is the “accidental Markdown syntax” problem, common with Markdown syntax like the italics syntax interacting with talking about char *s. • What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this ﬁgure substantially.
137 Zulip Documentation, Release 1.4.0
Both of these are minor issues for most products using Markdown, but they are major problems in the instant messaging context, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’s Markdown strategy is based on the principles of giving users the power they need to express complicated ideas in a chat context while minimizing those two error rates.
31.2 Zulip’s Changes to Markdown
Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify / disable may already be non-standard.
31.2.1 Basic syntax
• Enable ‘nl2br extension: this means one newline creates a line break (not paragraph break). • Disable italics entirely. This resolves an issue where people were using * and _ and hitting it by mistake too often. E.g. with stock Markdown You should use char * instead of void * there would trigger italics. • Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something) • Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having no escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.
31.2.2 Lists
• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it • Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not included in a code block) • Disable ordered list syntax: it automatically renumbers, which can be really confusing when sending a numbered list across multiple messages.
31.2.3 Links
• Enable auto-linkiﬁcation, both for http://... and guessing at things like t.co/foo. • Force links to be absolute. [foo](google.com) will go to http://google.com, and not http://zulip.com/google.com which is the default behavior. • Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window • Disable link-by-reference syntax, [foo][bar] ... [bar]: http://google.com
31.2.4 Code
• Enable fenced code block extension, with syntax highlighting • Disable line-numbering within fenced code blocks – the 
 output confused our web client code.
138 Chapter 31. Markdown implementation Zulip Documentation, Release 1.4.0
31.2.5 Other
• Disable headings, both # foo and == foo == syntax: they don’t make much sense for chat messages. • Disabled images. • Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integra- tions. • We added the ~~~ quote block quote syntax.
31.2. Zulip’s Changes to Markdown 139 Zulip Documentation, Release 1.4.0
140 Chapter 31. Markdown implementation CHAPTER 32
Static asset pipeline
This page documents additional information that may be useful when developing new features for Zulip that require front-end changes. For a more general overview, see the new-feature-tutorial. The code-style documentation also has relevant information about how Zulip’s code is structured.
32.1 Primary build process
Most of the existing JS in Zulip is written in IIFE-wrapped modules, one per ﬁle in the static/js directory. When running Zulip in development mode, each ﬁle is loaded seperately. In production mode (and when creating a release tarball using tools/build-release-tarball), JavaScript ﬁles are concatenated and miniﬁed. If you add a new JavaScript ﬁle, it needs to be speciﬁed in the JS_SPECS dictionary deﬁned in zproject/settings.py to be included in the concatenated ﬁle.
32.2 Webpack/CommonJS modules
New JS written for Zulip can be written as CommonJS modules (bundled using webpack, though this will taken care of automatically whenever run-dev.py is running). (CommonJS is the same module format that Node uses, so see the Node documentation for more information on the syntax.) Beneﬁts of using CommonJS modules over the IIFE module approach: • namespacing/module boilerplate will be added automatically in the bundling process • dependencies between modules are more explicit and easier to trace • no separate list of JS ﬁles needs to be maintained for concatenation and miniﬁcation • third-party libraries can be more easily installed/versioned using npm • running the same code in the browser and in Node for testing is simpliﬁed (as both environments use the same module syntax) The entry point ﬁle for the bundle generated by webpack is static/js/src/main.js. Any modules you add will need to be required from this ﬁle (or one of its dependencies) in order to be included in the script bundle.
32.3 Adding static ﬁles
To add a static ﬁle to the app (JavaScript, CSS, images, etc), ﬁrst add it to the appropriate place under static/.
141 Zulip Documentation, Release 1.4.0
• Third-party ﬁles should all go in static/third/. Tag the commit with “[third]” when adding or modifying a third-party package. • Our own JS lives under static/js; CSS lives under static/styles. • JavaScript and CSS ﬁles are combined and miniﬁed in production. In this case all you need to do is add the ﬁlename to PIPELINE[‘STYLESHEET’] or JS_SPECS in zproject/settings.py. (If you plan to only use the JS/CSS within the app proper, and not on the login page or other standalone pages, put it in the ‘app’ category.) If you want to test miniﬁed ﬁles in development, look for the PIPELINE_ENABLED = line in zproject/settings.py and set it to True – or just set DEBUG = False. Note that static/html/{400,5xx}.html will only render properly if miniﬁcation is enabled, since they hard- code the path static/min/portico.css.
142 Chapter 32. Static asset pipeline CHAPTER 33
Schema Migrations
Zulip uses the standard Django system for doing schema migrations. There is some example usage in the new feature tutorial. This page documents some important issues related to writing schema migrations. • Large tables: For large tables like Message and UserMessage, you want to take precautions when adding columns to the table, performing data backﬁlls, or building indexes. We have a zerver/lib/migrate.py library to help with adding columns and backﬁlling data. For building indexes on these tables, we should do this using SQL with postgres’s CONCURRENTLY keyword. • Numbering conﬂicts across branches: If you’ve done your schema change in a branch, and meanwhile another schema change has taken place, Django will now have two migrations with the same number. To ﬁx this, you need to renumber your migration(s), ﬁx up the “dependencies” entries in your migration(s), and rewrite your git history as needed. There is a tutorial here that walks you though that process.
143 Zulip Documentation, Release 1.4.0
144 Chapter 33. Schema Migrations CHAPTER 34
HTML and CSS
34.1 Zulip CSS organization
The Zulip application’s CSS can be found in the static/styles/ directory. Zulip uses Bootstrap as its main third-party CSS library. Zulip currently does not use any CSS preprocessors, and is organized into several ﬁles. For most pages, the CSS is combined into a single CSS ﬁle by the static asset pipeline, controlled by the PIPELINE_CSS code in zproject/settings.py. The CSS ﬁles are: • portico.css - Main CSS for logged-out pages • pygments.css - CSS for Python syntax highlighting • activity.css - CSS for the activity app • fonts.css - Fonts for text in the Zulip app • static/styles/thirdparty-fonts.css - Font Awesome (used for icons) The CSS for the Zulip web application UI is primarily here: • settings.css - CSS for the Zulip settings and administration pages • zulip.css - CSS for the rest of the Zulip logged-in app • media.css - CSS for media queries (particularly related to screen width) We are in the process of splitting zulip.css into several more ﬁles; help with that project is very welcome!
34.2 Editing Zulip CSS
If you aren’t experienced with doing web development and want to make CSS changes, we recommend reading the excellent Chrome web inspector guide on editing HTML/CSS, especially the section on CSS to learn about all the great tools that you can use to modify and test changes to CSS interactively in-browser (without even having the reload the page!).
145 Zulip Documentation, Release 1.4.0
34.3 CSS Style guidelines
34.3.1 Avoid duplicated code
Without care, it’s easy for a web application to end up with thousands of lines of duplicated CSS code, which can make it very difﬁcult to understand the current styling or modify it. We would very much like to avoid such a fate. So please make an effort to reuse existing styling, clean up now-unused CSS, etc., to keep things maintainable.
34.3.2 Be consistent with existing similar UI
Ideally, do this by reusing existing CSS declarations, so that any improvements we make to the styling can improve all similar UI elements.
34.3.3 Use clear, unique names for classes and object IDs
This makes it much easier to read the code and use git grep to ﬁnd where a particular class is used.
34.4 Validating CSS
When changing any part of the Zulip CSS, it’s important to check that the new CSS looks good at a wide range of screen widths, from very wide screen (e.g. 1920px) all the way down to narrow phone screens (e.g. 480px). For complex changes, it’s deﬁnitely worth testing in a few different browsers to make sure things look the same.
146 Chapter 34. HTML and CSS CHAPTER 35
Full-text search
Zulip supports full-text search, which can be combined arbitrarily with Zulip’s full suite of narrowing operators. By default, it only supports English text, but there is an experimental PGroonga integration that provides full-text search for all languages. The user interface and feature set for Zulip’s full-text search is documented in the “Search help” documentation section in the Zulip app’s gear menu.
35.1 The default full-text search implementation
Zulip’s uses PostgreSQL’s built-in full-text search feature, with a custom set of English stop words to improve the quality of the search results. We use a small extension, tsearch_extras, for highlighting of the matching words. There is some discussion of remov- ing this extension, at least as an option, so that Zulip can be used with database-as-a-service platforms. In order to optimize the performance of delivering messages, the full-text search index is updated for newly sent messages in the background, after the message has been delivered. This background updating is done by puppet/zulip/files/postgresql/process_fts_updates, which is usually deployed on the database server, but could be deployed on an application server instead.
35.2 An optional full-text search implementation
See the option PGroonga pull request for details on the status of the PGroonga integration.
147 Zulip Documentation, Release 1.4.0
148 Chapter 35. Full-text search CHAPTER 36
Translating Zulip
Zulip has full support for unicode, so you can already use your preferred language everywhere in Zulip. To make Zulip even better for users around the world, the Zulip UI is being translated into a number of major languages, including Spanish, German, French, Chinese, Russian, and Japanese, with varying levels of progress. If you speak a language other than English, your help with translating Zulip would be greatly appreciated! If you’re interested in contributing translations to Zulip, join the Zulip project on Transifex and ask to join any lan- guages you’d like to contribute to (or add new ones). Transifex’s notiﬁcation system sometimes fails to notify the maintainers when you ask to join a project, so please send a quick email to [email protected] when you request to join the project or add a language so that we can be sure to accept your request to contribute.
36.1 Setting Default Language in Zulip
Zulip allows you to set the default language through the settings page under ‘Display Settings’ section.
36.2 Translation Resource Files
All the translation magic happens through resource ﬁles which hold the translated text. Backend resource ﬁles are located at static/locale//LC_MESSAGES/django.po, while frontend resource ﬁles are lo- cated at static/locale//translations.json. These ﬁles are uploaded to Transifex using tx push, where they can be translated. Once translated, they are downloaded back into the codebase using tx pull.
36.3 Transifex Conﬁg
The conﬁg ﬁle that maps the resources from Zulip to Transifex is located at .tx/config. Django recognizes zh_CN instead of zh-HANS for simpliﬁed Chinese language (this is ﬁxed in Django 1.9). This idiosyncrasy is also handled in the Transifex conﬁg ﬁle.
36.4 Translation Process
The end-to-end process to get the translations working is as follows: 1. Mark the strings for translations (see sections for backend and frontend translations for details on this).
149 Zulip Documentation, Release 1.4.0
2. Create JSON formatted resource ﬁles using the python manage makemessages command. This com- mand will create a resource ﬁle called translations.json for frontend and django.po for backend for every language under static/locale. The location for frontend resource ﬁle can be changed by passing an argument to the command (see the help for the command for further details). However, make sure that the location is publicly accessible since frontend ﬁles are loaded through XHR in the frontend which will only work with publicly accessible resources. The makemessages command is idempotent in that: • It will only delete singular keys in the resource ﬁle when they are no longer used in Zulip code. • It will only delete plural keys (see below for the documentation on plural translations) when the corre- sponding singular key is absent. • It will not override the value of a singular key if that value contains a translated text. 3. Upload the resource ﬁles to Transifex using the tx push -s -a command. 4. Download the updated resource ﬁles from Transifex using the tx pull -a command. This command will download the resource ﬁles from Transifex and replace your local resource ﬁles with them.
36.5 Backend Translations
All user-facing text in the Zulip UI should be generated by an HTML template so that it can be translated. Zulip uses two types of templates: backend templates (powered by the Jinja2 template engine, though the original Django template engine is still supported) and frontend templates (powered by Handlebars). To mark a string for translation in the Jinja2 and Django template engines, you can use the _() function in the templates like this:
{{ _("English text") }}
If a string contains both a literal string component and variables, you can use a block translation, which makes use of placeholders to help translators to translate an entire sentence. To translate a block, Jinja2 uses the trans tag while Django uses the blocktrans tag. So rather than writing something ugly and confusing for translators like this:
# Don't do this! {{ _("This string will have") }} {{ value }} {{ _("inside") }}
You can instead use:
# Jinja2 style {% trans%}This string will have {{ value }} inside.{% endtrans%} # Django style {% blocktrans%}This string will have {{ value }} inside.{% endblocktrans%}
Zulip expects all the error messages to be translatable as well. To ensure this, the error message passed to json_error and JsonableError should always be a literal string enclosed by _() function, e.g: json_error(_('English Text')) JsonableError(_('English Text'))
To ensure we always internationalize our JSON errors messages, the Zulip linter (tools/lint-all) checks for correct usage.
150 Chapter 36. Translating Zulip Zulip Documentation, Release 1.4.0
36.6 Frontend Translations
Zulip uses the i18next library for frontend translations. There are two types of ﬁles in Zulip frontend which can hold translatable strings, JavaScript code ﬁles and Handlebar templates. To mark a string translatable in JavaScript ﬁles pass it to the i18n.t function. i18n.t('English Text', context); i18n.t('English text with a __variable__',{'variable':'Variable value'});
Note: In the second example above, instead of enclosing the variable with handlebars, {{ }}, we enclose it with __ because we need to differentiate the variable from the Handlebar tags. The symbol which is used to enclose the variables can be changed in /static/js/src/main.js. i18next also supports plural translations. To support plurals make sure your resource ﬁle contatins the related keys:
{ "en":{ "translation":{ "key":"item", "key_plural":"items", "keyWithCount":"__count__ item", "keyWithCount_plural":"__count__ items" } } }
With this resource you can show plurals like this: i18n.t('key', {count:0});// output:'items' i18n.t('key', {count:1});// output:'item' i18n.t('key', {count:5});// output:'items' i18n.t('key', {count: 100});// output:'items' i18n.t('keyWithCount', {count:0});// output:'0 items' i18n.t('keyWithCount', {count:1});// output:'1 item' i18n.t('keyWithCount', {count:5});// output:'5 items' i18n.t('keyWithCount', {count: 100});// output:'100 items'
For further reading on plurals, read the ofﬁcial documentation. To mark the strings as translatable in the Handlebar templates, Zulip registers two Handlebar helpers. The syntax for simple strings is:
{{t'English Text'}}
The syntax for block strings or strings containing variables is:
{{tr context}} Block of English text. {{/tr}} var context={'variable':'variable value'}; {{tr context}} Block of English text with a __variable__. {{/tr}}
The rules for plurals are same as for JavaScript ﬁles. You just have to declare the appropriate keys in the resource ﬁle and then include the count in the context.
36.6. Frontend Translations 151 Zulip Documentation, Release 1.4.0
36.7 Testing Translations
First of all make sure that you have compiled the translation strings using python manage.py compilemessages. Django ﬁgures out the effective language by going through the following steps: 1. It looks for the language code in the url. 2. It looks for the LANGUGE_SESSION_KEY key in the current user’s session. 3. It looks for the cookie named ‘django_language’. You can set a different name through LAN- GUAGE_COOKIE_NAME setting. 4. It looks for the Accept-Language HTTP header in the HTTP request. Normally your browser will take care of this. The easiest way to test translations is through the i18n urls e.g. if you have German translations available you can access the German version of a page by going to /de/path_to_page. To test translations using other methods you will need an HTTP client library like requests, cURL or urllib. Here is a sample code to test Accept-Language header using requests:
 import requests headers={"Accept-Language":"de"} response= requests.get("http://localhost:9991/login/", headers=headers) print(response.content)
152 Chapter 36. Translating Zulip CHAPTER 37
Logging and Performance Debugging
It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. The messages will look similar to:
2016-05-20 14:50:22,056 INFO 127.0.0.1 GET 302 528ms (db: 1ms/1q)
˓→(+start: 123ms) / (unauth via ?) [20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0 2016-05-20 14:50:22,272 INFO 127.0.0.1 GET 200 124ms (db: 3ms/2q) /
˓→login/ (unauth via ?) 2016-05-20 14:50:26,333 INFO 127.0.0.1 POST 302 37ms (db: 6ms/7q) /
˓→accounts/login/local/ (unauth via ?) [20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0 2016-05-20 14:50:26,538 INFO 127.0.0.1 GET 200 12ms (db: 1ms/2q)
˓→(+start: 53ms) /api/v1/events [1463769771:0/0] ([email protected] via internal) 2016-05-20 14:50:26,657 INFO 127.0.0.1 GET 200 10ms (+start: 8ms) /api/
˓→v1/events [1463769771:0/0] ([email protected] via internal) 2016-05-20 14:50:26,959 INFO 127.0.0.1 GET 200 588ms (db: 26ms/21q) /
˓→[1463769771:0] ([email protected] via website)
The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status code, time to process, (optional perf data details, e.g. database time/queries, memcached time/queries, Django process startup time, markdown processing time, etc.), URL, and “email via client” showing user account involved (if logged in) and the type of client they used (“web”, “Android”, etc.).
153 Zulip Documentation, Release 1.4.0
154 Chapter 37. Logging and Performance Debugging CHAPTER 38
Documentation
These docs are written in Commonmark Markdown with a small bit of rST. We’ve chosen Markdown because it is easy to write. The docs are served in production at zulip.readthedocs.io. If you want to build the documentation locally (e.g. to test your changes), the dependencies are automatically installed as part of Zulip development environment provisioning, and you can build the documentation using: cd docs/ make html and then opening file:///path/to/zulip/docs/_build/html/index.html in your browser (you can also use e.g. firefox docs/_build/html/index.html from the root of your Zulip checkout). You can also usually test your changes by pushing a branch to GitHub and looking at the content on the GitHub web UI, since GitHub renders Markdown. When editing dependencies for the Zulip documentation, you should edit requirements/docs.txt (which is used by ReadTheDocs to build the documentation quickly, without installing all of Zulip’s dependencies).
155 Zulip Documentation, Release 1.4.0
156 Chapter 38. Documentation CHAPTER 39
Indices and tables
• genindex • modindex • search
157