Kuma Documentation Release latest

Mozilla

October 30, 2020

Contents

1 Development 3

2 Getting started 5 2.1 Installation...... 5 2.2 Development...... 10 2.3 Troubleshooting...... 24 2.4 The Kuma test suite...... 26 2.5 Client-side testing...... 28 2.6 Feature toggles...... 30 2.7 Celery and async tasks...... 33 2.8 Search...... 35 2.9 Localization...... 36 2.10 CKEditor...... 39 2.11 Getting Data...... 41 2.12 Documentation...... 45 2.13 Docker...... 45 2.14 Deploying MDN...... 46 2.15 Rendering Documents...... 50 2.16 Building, collecting, and serving assets...... 63

i ii Kuma Documentation, Release latest

Kuma is the platform that powers MDN (developer.mozilla.org)

Contents 1 Kuma Documentation, Release latest

2 Contents CHAPTER 1

Development

Code https://github.com/mdn/kuma Issues P1 Bugs (to be fixed in current or next sprint) P2 Bugs (to be fixed in 180 days) All developer.mozilla.org bugs Pull Request Queues Dev Docs https://kuma.readthedocs.io/en/latest/installation.html CI Server https://travis-ci.com/mdn/kuma Forum https://discourse.mozilla.org/c/mdn Matrix #mdn room Servers What’s Deployed on MDN? https://developer.allizom.org/ (stage) https://developer.mozilla.org/ (prod)

3 Kuma Documentation, Release latest

4 Chapter 1. Development CHAPTER 2

Getting started

Want to help make MDN great? Our contribution guide lists some good first projects and offers direction on submitting code. Contents:

2.1 Installation

Kuma uses Docker for local development and integration testing, and we are transitioning to Docker containers for deployment as well. Current Status of Dockerization: • Kuma developers are using Docker for daily development and maintenance tasks. Staff developers primarily use Docker for Mac. Other staff members and contributors use Docker’s Ubuntu packages. • The development environment can use a lot of resources. On Docker for Mac, the environment runs well with 6 CPUs and 10 GB of memory dedicated to Docker. It can be run successfully on 2 CPUs and 2 GB of memory. • The Docker development environment is evolving rapidly, to address issues found during development and to move toward a containerized design. You may need to regularly reset your environment to get the current changes. • The Docker development environment doesn’t fully support a ‘production-like’ environment. For example, we don’t have a documented configuration for running with an SSL connection. • When the master branch is updated, the kuma_base image is refreshed and published to DockerHub. This image contains system packages and third-party libraries. • Our TravisCI builds include a target that build Docker containers and runs the tests inside. • Our Jenkins server builds and publishes Docker images, and runs integration tests using Docker. • We are documenting tips and tricks on the Troubleshooting page. • Feel free to ask for help on Matrix at #mdn or on discourse.

2.1.1 Docker setup

1. Install the Docker platform, following Docker’s instructions for your operating system, such as Docker for Mac for MacOS, or for your Linux distribution. Non-Linux users should increase Docker’s memory limits (Windows, macOS) to at least 4 GB, as the default of 2 GB is insufficient.

5 Kuma Documentation, Release latest

Linux users will also want to install Docker Compose and follow post-install instructions to confirm that the development user can run Docker commmands. To confirm that Docker is installed correctly, run: docker run hello-world

If you find any error using docker commands without sudo visit using docker as non-root user. 2. Clone the kuma Git repository, if you haven’t already: git clone --recursive https://github.com/mdn/kuma.git

If you think you might be submitting pull requests, consider forking the repository first, and then cloning your fork of it. 3. Ensure you are in the existing or newly cloned kuma working copy: cd kuma

4. Initialize and customize .env: cp.env-dist.dev.env vim.env # Or your favorite editor

Linux users should set the UID parameter in .env (i.e. change #UID=1000 to UID=1000) to avoid file permission issues when mixing docker-compose and docker commands. MacOS users do not need to change any of the defaults to get started. Note that there are settings in this file that can be useful when debug- ging, however. 5. Pull the Docker images and build the containers: docker-compose pull docker-compose build

(The build command is effectively a no-op at this point because the pull command just downloaded pre-built docker images.) 6. Start the containers in the background: docker-compose up -d

2.1.2 Load the sample database

Download the sample database with either of the following wget or curl (installed by default on macOS) commands: wget -N https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz curl -O https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz

Next, upload that sample database into the Kuma web container with: docker-compose exec web bash -c "zcat mdn_sample_db.sql.gz | ./manage.py dbshell"

(This command can be adjusted to restore from an uncompressed database, or directly from a mysqldump command.) Then run the following command: docker-compose exec web ./manage.py migrate

This will ensure the sample database is in sync with your version of Kuma.

6 Chapter 2. Getting started Kuma Documentation, Release latest

2.1.3 Compile locales

Localized string databases are included in their source form, and need to be compiled to their binary form: docker-compose exec web make localecompile

Dozens of lines of warnings will be printed: cd locale; ./compile-mo.sh . ./af/LC_MESSAGES/django.po:2: warning: header field 'PO-Revision-Date' still has the initial default value ./af/LC_MESSAGES/django.po:2: warning: header field 'Last-Translator' still has the initial default value ... ./zu/LC_MESSAGES/javascript.po:2: warning: header field 'PO-Revision-Date' still has the initial default value ./zu/LC_MESSAGES/javascript.po:2: warning: header field 'Last-Translator' still has the initial default value

Warnings are OK, and will be fixed as translators update the strings on Pontoon. If there is an error, the output will end with the error, such as: ./az/LC_MESSAGES/django.po:263: 'msgid' and 'msgstr' entries do not both end with '\n' msgfmt: found 1 fatal error

These need to be fixed by a Kuma developer. Notify them in the #mdn Matrix room or open a bug. You can continue with installation, but non-English locales will not be localized.

2.1.4 Generate static assets

Static assets such as CSS and JS are included in source form, and need to be compiled to their final form: docker-compose exec web make build-static

A few thousand lines will be printed, like: ## Generating JavaScript translation catalogs ## processing language en_US processing language af processing language ar ... ## Compiling (Sass), collecting, and building static files ## Copying '/app/kuma/static/img/embed/promos/survey.svg' Copying '/app/kuma/static/styles/components/home/column-callout.scss' Copying '/app/build/locale/jsi18n/fy-NL/javascript.js' ... Post-processed 'build/styles/editor-locale-ar.css' as 'build/styles/editor-locale-ar.css' Post-processed 'build/styles/locale-ln.css' as 'build/styles/locale-ln.css' Post-processed 'build/styles/editor-locale-pt-BR.css' as 'build/styles/editor-locale-pt-BR.css' .... 1870 static files copied to '/app/static', 125 post-processed.

2.1.5 Visit the homepage

Open the homepage at http://localhost.org:8000 . You’ve installed Kuma!

2.1.6 Create an admin user

Many Kuma settings require access to the Django admin, including configuring social login. It is useful to create an admin account with password access for local development.

2.1. Installation 7 Kuma Documentation, Release latest

If you want to create a new admin account, use createsuperuser: docker-compose exec web ./manage.py createsuperuser

This will prompt you for a username, email address (a fake address like [email protected] will work), and a password. If your database has an existing account that you want to use, run the management command. Replace YOUR_USERNAME with your username and YOUR_PASSWORD with your password: docker-compose run --rm web ./manage.py ihavepower YOUR_USERNAME \ --password YOUR_PASSWORD

With a password-enabled admin account, you can log into Django admin at http://localhost.org:8000/admin/login

2.1.7 Update the Sites section

1. After logging in to the Django admin (an alternative is using the login test-super with password test-password), scroll down to the Sites section. 2. Click on “Change”. 3. Click on the entry that says localhost:8000. 4. Change both the domain and display name from localhost:8000 to localhost.org:8000. 5. Click “Save”.

2.1.8 Enable GitHub/Google authentication (optional)

Since Google’s OAuth requires us to use a valid top-level-domain, we’re going to use http://localhost.org:8000 as an example for every URL here. To automate setting Django up for social auth you can run docker-compose exec web ./manage.py configure_social_auth and follow its steps (and ignore the rest of this section). If you want to do it manually, follow these steps: To enable GitHub authentication, you’ll need to register an OAuth application on GitHub, with settings like: • Application name: MDN Development for (). • Homepage URL: http://localhost.org:8000/. • Application description: My own GitHub app for MDN! • Authorization callback URL: http://localhost.org:8000/users/github/login/callback/. To enable Google authentication, you’ll need to first create an API project on Google. After that we’ll need to configure credentials for that project with settings like: • Name: MDN Development for (). • Authorized JavaScript origins: http://localhost.org:8000 • Authorized redirect URIs: http://localhost.org:8000/users/google/login/callback/ As an admin user, add a django-allauth social app for both GitHub and Google do the following: • Provider: GitHub/Google. • Name: MDN Development.

8 Chapter 2. Getting started Kuma Documentation, Release latest

• Client id: . • Secret key: . • Sites: Move locahost:8000 from “Available sites” to “Chosen sites”. locahost:8000 needs to either have ID 1 or SITE_ID=1 has to be set in .env to its actual ID. You’ll also need to set DOMAIN=localhost.org (no port!) there. Your hosts file should contain the following lines: 127.0.0.1 localhost demos localhost.org wiki.localhost.org 255.255.255.255 broadcasthost ::1 localhost demos localhost.org wiki.localhost.org

Now you can sign in with GitHub. To associate your password-only admin account with GitHub: 1. Login with your password at http://localhost.org:8000/admin/login. 2. Go to the Homepage at https://developer.mozilla.org/en-US/. 3. Click your username at the top to view your profile. 4. Click Edit to edit your profile. 5. Under My Profiles, click Use your GitHub account to sign in. To create a new account with GitHub, use the regular “Sign in” widget at the top of any page. With social accounts are enabled, you can disable the admin password in the Django shell: docker-compose exec web ./manage.py shell_plus >>> me = User.objects.get(username='admin_username') >>> me.set_unusable_password() >>> me.save() >>> exit()

2.1.9 Enable Stripe payments (optional)

1. Go to https://dashboard.stripe.com and create a Stripe account (if you don’t have one already). 2. Go to https://dashboard.stripe.com/apikeys and copy both the publishable and secret key into your .env file. The respective config keys are STRIPE_PUBLIC_KEY and STRIPE_SECRET_KEY. 3. Go to https://dashboard.stripe.com/test/subscriptions/products and create a new product and plan. 4. Once created copy the plan ID and also put it into .env as STRIPE_PLAN_ID. Unless you set a custom ID it should start with plan_. If you’re using Stripe in testing mode you can also get test numbers from this site: https://stripe.com/docs/testing#cards Testing Stripe’s hooks locally requires setting up a tunneling service, like ngrok (https://ngrok.com). You should then set CUSTOM_WEBHOOK_HOSTNAME to the hostname you get from your tunneling service, e.g. for ngrok it might be https://203ebfab.ngrok.io After kuma has started you will have a webhook configured in stripe. You can view it on Stripe’s dashboard: https://dashboard.stripe.com/test/webhooks Note that with the free tier a restart of ngrok gives you a new hostname, so you’ll have to change the config again and restart the server (or manually change the webhook in Stripe’s dashboard).

2.1. Installation 9 Kuma Documentation, Release latest

2.1.10 Enable Sendinblue email integration

1. Create a Sendinblue account over at https://www.sendinblue.com (you can skip a lot of the profile set-up, look for skip in the upper right). 2. Get your v3 API key at https://account.sendinblue.com/advanced/api 3. Create a list at https://my.sendinblue.com/lists/new/parent_id/1 4. Add the sendinblue config keys to your .env, the keynames are SENDINBLUE_API_KEY and SENDINBLUE_LIST_ID

2.1.11 Interact with the Docker containers

The current directory is mounted as the /app folder in the web and worker containers. Changes made to your local directory are usually reflected in the running containers. To force the issue, the containers for specified services can be restarted: docker-compose restart web worker

You can connect to a running container to run commands. For example, you can open an interactive shell in the web container: docker-compose exec web /bin/bash make bash # Same command, less typing

To view the logs generated by a container: docker-compose logs web

To continuously view logs from all containers: docker-compose logs -f

To stop the containers: docker-compose stop

If you have made changes to the .env or /etc/hosts file, it’s a good idea to run: docker-compose stop docker-compose up

For further information, see the Docker documentation, such as the Docker Overview and the documentation for your operating system. You can try Docker’s guided tutorials, and apply what you’ve learned on the Kuma Docker environment.

2.2 Development

2.2.1 Basic Docker usage

Edit files as usual on your host machine; the current directory is mounted via Docker host mounting at /app within various Kuma containers. Useful docker sub-commands:

10 Chapter 2. Getting started Kuma Documentation, Release latest

docker-compose exec web bash # Start an interactive shell docker-compose logs web # View logs from the web container docker-compose logs -f # Continuously view logs from all containers docker-compose restart web # Force a container to reload docker-compose stop # Shutdown the containers docker-compose up -d # Start the containers docker-compose rm # Destroy the containers

There are make shortcuts on the host for frequent commands, such as: make up # docker-compose up -d make bash # docker-compose exec web bash make shell_plus # docker-compose exec web ./manage.py shell_plus

Run all commands in this doc in the web service container after make bash.

2.2.2 Running Kuma

When the Docker container environment is started (make up or similar), all of the services are also started. The development instance is available at http://localhost:8000.

2.2.3 Running the tests

One way to confirm that everything is working, or to pinpoint what is broken, is to run the test suite.

Django tests

Run the Django test suite: make test

For more information, see the test documentation.

Functional tests

To run the functional tests, see Client-side Testing.

Kumascript tests

If you’re changing Kumascript, be sure to run its tests too. See https://github.com/mdn/kumascript.

2.2.4 Front-end development

Assets are processed in several steps by Django and django-pipeline: • Front-end localization libraries and string catalogs are generated based on gettext calls in Javascript files. • Sass source files are compiled to plain CSS with node-sass. • Assets are collected to the static/ folder. • CSS, JS, and image assets are included in templates using the {% stylesheet %}, {% javascript %}, and {% static %} Jinja2 macros.

2.2. Development 11 Kuma Documentation, Release latest

In production mode (DEBUG=False), there is additional processing. See Generating production assets for more information. To rebuild the front-end localization libraries: make compilejsi18n

To rebuild CSS, JS, and image assets: make collectstatic

To do both: make build-static

Compiling JS on the host system with webpack

For a quicker iteration cycle while developing the frontend app you can run: npm i npm run webpack:dev

This watches the react frontend and rebuilds both the web and SSR bundle when changes occur. It rebuilds only what has changed and also restarts the SSR server. It serves React in development mode which yields more explicit warnings and allows you to use tools such as the React DevTools. You can also run it in production mode: npm run webpack:prod

Style guide and linters

There is an evolving style guide at https://mdn.github.io/mdn-fiori/, sourced from https://github.com/mdn/mdn-fiori. Some of the style guidelines are enforced by linters. To run stylelint on all .scss files: npm run stylelint

To run eslint on .js files: npm run eslint

Bundle Analyze

To get an idea about the size distribution of our react codebase you can run: npm run webpack:analyze

This will open an interactively explorable report in your browser.

Add a React page

Let’s say we are adding a new Example page to the dashboards section on Kuma. We’d like the pathname to be /dashboards/example/. First we need to set up a few things in Django:

12 Chapter 2. Getting started Kuma Documentation, Release latest

1. In kuma/dashboards/jinja2/dashboards/, create a new Jinja template that our view will render called example.html. Paste the following code in example.html to get started: {% extends "react_base.html" %}

{% block title %}{{ page_title(_('Example Page')) }}{% endblock %}

{% block content %}This works!{% endblock %}

{# This overrides the Jinja footer, since we will be using the React footer instead. Without this line, two footers will render. #} {% block footer %}{% endblock %}

2. In kuma/dashboards/views.py, render the Jinja template we just created: def example(request): return render(request,"dashboards/example.html")

3. In kuma/dashboards/urls.py, add a new url that will render our view. This code will change slightly depending on your url and file structure: # Import the view

# (This next line most likely will already be there, if not, add it to the top of the file.) from. import views

...

re_path(r"^example/$", views.example, name="dashboards.example"),

4. We are done with the Django part! In your browser, go to http://wiki.localhost.org:8000/en- US/dashboards/example/ and you should see a page that says “This works!” 5. Now we can set up the React part. In kuma/dashboards/jinja2/dashboards/example.html, we can remove the “This works!” text and add the following code between {% block title %} and {% block content %}: {# Pass `True` for initial data if you do not have any data to fetch. Otherwise, change `True` to `None` or whatever your initial data will be. #} {% block document_head %} {{ render_react("SPA", request.LANGUAGE_CODE, request.get_full_path(), True)|safe }} {% endblock %}

6. So your kuma/dashboards/jinja2/dashboards/example.html file should now look something like this: {% extends "react_base.html" %}

{% block title %}{{ page_title(_('Example Page')) }}{% endblock %}

{# Pass `True` for initial data if you do not have any data to fetch (i.e. static page). #} {# Otherwise, change `True` to `None` or whatever your initial data will be. #} {% block document_head %} {{ render_react("SPA", request.LANGUAGE_CODE, request.get_full_path(), True)|safe }} {% endblock %}

{% block content %} {% endblock %}

{# This overrides the Jinja footer, since we will be using the React footer instead. Without this line, two footers will render. #} {% block footer %}{% endblock %}

2.2. Development 13 Kuma Documentation, Release latest

7. In kuma/javascript/src, if there is not a kuma/javascript/dashboards folder already, add a folder called dashboards. 8. In kuma/javascript/src/dashboards, create a file called example.jsx. Add the following lines as a starter: // @flow import * as React from 'react';

import { gettext } from '../l10n.js'; import A11yNav from '../a11y/a11y-nav.jsx'; import Header from '../header/header.jsx'; import Footer from '../footer.jsx'; import Route from '../route.js';

type ExampleRouteParams = { locale: string };

export default function ExamplePage() { return ( <>

{gettext('Hello!')}

); }

const BASEURL = typeof window !== 'undefined' && window.location ? window.location.origin : 'http://ssr.hack';

export class ExampleRoute extends Route { locale: string;

constructor(locale: string) { super(); this.locale = locale; }

getComponent() { return ExamplePage; }

match(url: string): ?ExampleRouteParams { const path = new URL(url, BASEURL).pathname; const examplePath = `/${this.locale}/dashboards/example/`; const regex = new RegExp(examplePath, 'g');

if (regex.test(path)) { return { locale: this.locale }; } return null; }

14 Chapter 2. Getting started Kuma Documentation, Release latest

}

9. We need to tell the single-page-app component about our new Route component, ExampleRoute. In kuma/javascript/src/single-page-app.jsx, at the top, add: import { ExampleRoute } from './dashboards/example.jsx';

10. A few lines down in the same file, where you see const routes = [ ... ], add ExampleRoute, so it should look like this: const routes = [ ... new ExampleRoute(locale) ];

11. We are done! Go to /dashboards/example/ in your browser and you should see a header, footer, and the words, “Hello!”

React page with initial data

If we have initial data we’d like to pass to our React component, we can do so by updating the 4th argument (document_data) in the render_react function call. This data will then be available as a prop on the Re- act side. For a real-life example, search for document_data in kuma/wiki/views/document.py. 1. Continuing our example from above, go back to kuma/dashboards/jinja2/dashboards/example.html. In the call to render_react, change True to your initial data. For our purposes, initial data will be {"content": "Goodbye!"}. The full line should look like:

{{ render_react("SPA", request.LANGUAGE_CODE, request.get_full_path(), {"content":"Goodbye!"})|safe }}

2. In kuma/javascript/src/dashboards/example.jsx, import the RouteComponentProps type by making the change below: import Route from '../route.js';

↑ to

import Route, { type RouteComponentProps } from '../route.js';

3. In the same file, update the ExamplePage function to accept a data parameter. The function declaration should now look like this: export default function ExamplePage({ data }: RouteComponentProps) { ... }

4. Now we can access the initial data from Step 1. Change {gettext("Hello!)} to {gettext(data.content)}. Refresh the page and the “Hello!” text should be replaced by “Goodbye!”

React page using fetch()

In some cases, you may choose to fetch data asynchronously when the page renders. 1. Continuing our example from above, go back to kuma/dashboards/jinja2/dashboards/example.html. In the call to render_react, change the 4th argument to None. The full line should look like:

2.2. Development 15 Kuma Documentation, Release latest

{{ render_react("SPA", request.LANGUAGE_CODE, request.get_full_path(), None)|safe }}

2. Refresh the page in the browser and you should now get an error. That is because we do not have a fetch() method yet, so let’s make one. In kuma/javascript/src/dashboards/example.jsx, ExampleRoute class, add this method (note: we are using /api/v1/whoami just to show a working ex- ample of an API call. Change this to reflect your API endpoint.): fetch(): Promise { return fetch('/api/v1/whoami').then(response => response.json());

}

3. Once the fetch completes, our data will be available as a data prop once again. In the ExamplePage function, replace the {gettext(data.content)} line with: {!data && gettext('Loading...')} {data && gettext(data.username)}

4. Your kuma/javascript/src/dashboards/example.jsx file should look like: // @flow import * as React from 'react';

import { gettext } from '../l10n.js'; import A11yNav from '../a11y/a11y-nav.jsx'; import Header from '../header/header.jsx'; import Footer from '../footer.jsx'; import Route, { type RouteComponentProps } from '../route.js';

type ExampleRouteParams = { locale: string };

export default function ExamplePage({ data }: RouteComponentProps) { return ( <>

{!data && gettext('Loading...')} {data && gettext(data.username)}

); }

const BASEURL = typeof window !== 'undefined' && window.location ? window.location.origin : 'http://ssr.hack';

export class ExampleRoute extends Route { locale: string;

constructor(locale: string) { super(); this.locale = locale; }

getComponent() {

16 Chapter 2. Getting started Kuma Documentation, Release latest

return ExamplePage; }

match(url: string): ?ExampleRouteParams { const path = new URL(url, BASEURL).pathname; const examplePath = `/${this.locale}/dashboards/example/`; const regex = new RegExp(examplePath, 'g');

if (regex.test(path)) { return { locale: this.locale }; } return null; }

fetch(): Promise { return fetch('/api/v1/whoami').then(response => response.json()); } }

5. Refresh the page in the browser, and you should now see your username if you are logged in. Or just the header and footer if you are logged out.

2.2.5 Database migrations

Apps are migrated using Django’s migration system. To run the migrations: ./manage.py migrate

If your changes include schema modifications, see the Django documentation for the migration workflow.

2.2.6 Coding conventions

See CONTRIBUTING.md for details of the coding style on Kuma. New code is expected to have test coverage. See the Test Suite docs for tips on writing tests.

2.2.7 Managing dependencies

Python dependencies

Kuma uses Poetry for dependency management. Poetry is configured in the pyproject.toml file at the root of the repository, and exact versions of dependencies (along with hashes) are stored in the poetry.lock file. Please refer to the Poetry docs on adding and updating dependencies. A few examples: • Use poetry update to update and re-lock all dependencies to their latest compatible versions, according to constraints in pyproject.toml. • Use poetry update to update only a single dependency to its latest compatible version, according to constraints in pyproject.toml. For example poetry update pytz.

2.2. Development 17 Kuma Documentation, Release latest

• To update a package to the very latest and not just what matches what’s currently in pyproject.toml, add @latest. For example poetry update pytz@latest. • Use poetry add to modify or add new entries inside of pyproject.toml, for example, poetry add django~2.2 or poetry add flake8@latest. • Use poetry lock to regenerate the poetry.lock, for example, after manually editing pyproject.toml. • Use poetry show to report on the project’s dependencies. In brief, update alters the lockfile, but does not modify entries within pyproject.toml. The add command changes both. You may wish to run these commands inside of Docker: docker-compose exec web poetry update --dry-run

Using Poetry directly on your host computer is also fine; the resulting pyproject.toml and poetry.lock files should be the same either way.

Front-end asset dependencies

Front-end dependencies are managed by Bower and checked into the repository. Follow these steps to add or upgrade a dependency: 1. On the host, update bower.json. 2. Start a root Docker container shell docker-compose run -u root web bash 3.( Docker only) In the root container shell, run: apt-get update apt-get install -y git npm install -g bower-installer bower-installer

4. On the host, prepare the dependency to be committed (git add path/to/dependency). Front-end dependencies that are not already managed by Bower should begin using this approach the next time they’re upgraded.

Front-end toolchain dependencies

The Front-end toolchain dependencies are managed by yarn, but not checked in to the repository. Follow these steps to add a dependency: docker-compose exec web bash Once you’re inside the Docker container, run: yarn add my-needed-new-lib Exit the container and the check in the changes to package.json and yarn.lock. To upgrade a dependency: docker-compose exec web bash Once you’re inside the Docker container, run: yarn upgrade-interactive –latest

18 Chapter 2. Getting started Kuma Documentation, Release latest

Use the interactive prompt. Exit the container and the check in the changes to package.json and yarn.lock.

2.2.8 Customizing with environment variables

Environment variables are used to change the way different components work. There are a few ways to change an environment variables: • Exporting in the shell, such as: export DEBUG=True; ./manage.py runserver

• A one-time override, such as: DEBUG=True ./manage.py runserver

• Changing the environment list in docker-compose.yml. • Creating a .env file in the repository root directory. One variable you may wish to alter for local development is DEBUG_TOOLBAR, which, when set to True, will enable the Django Debug Toolbar: DEBUG_TOOLBAR=True

Note that enabling the Debug Toolbar can severely impact response time, adding around 4 seconds to page load time.

2.2.9 Customizing number of workers

The docker-compose.yml in git comes with a default setting of 4 celery workers and 4 gunicorn workers. That’s pretty resource intensive since they prefork. To change the number of gunicorn and celery workers, consider setting this in your .env file: CELERY_WORKERS=2 GUNICORN_WORKERS=3

In that example, it will only start 2 celery workers and 3 gunicorn workers just for your environment.

2.2.10 Customizing the Docker environment

Running docker-compose will create and run several containers, and each container’s environment and settings are configured in docker-compose.yml. The settings are “baked” into the containers created by docker-compose up. To override a container’s settings for development, use a local override file. For example, the web service runs in a container with the default command “gunicorn -w 4 --bind 0.0.0.0:8000 --timeout=120 kuma.wsgi:application”. (The container has a name that begins with kuma_web_1_ and ends with a string of random hex digits. You can look up the name of your particular container with docker ps | grep kuma_web. You’ll need this container name for some of the commands described below.) A useful alternative for debugging is to run a single-threaded process that loads the Werkzeug debugger on exceptions (see docs for run- server_plus), and that allows for stepping through the code with a debugger. To use this alternative, create an override file docker-compose.override.yml: version: "2.1" services: web:

2.2. Development 19 Kuma Documentation, Release latest

command: ./manage.py runserver_plus 0.0.0.0:8000 stdin_open: true tty: true

This is similar to “docker run -it ./manage.py runserver_plus”, using all the other configuration items in docker-compose.yml. Apply the custom setting with: docker-compose up -d

You can then add pdb breakpoints to the code (import pdb; pdb.set_trace) and connect to the debugger with: docker attach

A similar method can be used to override environment variables in containers, run additional services, or make other changes. See the docker-compose documentation for more ideas on customizing the Docker environment.

2.2.11 Customizing the database

The database connection is defined by the environment variable DATABASE_URL, with this default: DATABASE_URL=mysql://root:kuma@mysql:3306/developer_mozilla_org

The format is defined by the dj-database-url project: DATABASE_URL=mysql://user:password@host:port/database

If you configure a new database, override DATABASE_URL to connect to it. To add an empty schema to a freshly created database: ./manage.py migrate

To connect to the database specified in DATABASE_URL, use: ./manage.py dbshell

2.2.12 Generating production assets

Setting DEBUG=False will put you in production mode, which adds aditional asset processing: • Javascript modules are combined into single JS files. • CSS and JS files are minifed and post-processed by cleancss and UglifyJS. • Assets are renamed to include a hash of contents by a variant of Django’s ManifestStaticFilesStorage. In production mode, assets and their hashes are read once when the server starts, for efficiency. Any changes to assets require rebuilding with make build-static and restarting the web process. To emulate production, and test compressed and hashed assets locally: 1. Set the environment variable DEBUG=False 2. Start (docker-compose up -d) your Docker services. 3. Run docker-compose run --rm -e DJANGO_SETTINGS_MODULE=kuma.settings.prod web make build-static. 4. Restart the web process using docker-compose restart web.

20 Chapter 2. Getting started Kuma Documentation, Release latest

2.2.13 Using secure cookies

To prevent error messages like “Forbidden (CSRF cookie not set.):”, set the environment variable: CSRF_COOKIE_SECURE= false

This is the default in Docker, which does not support local development with HTTPS.

2.2.14 Maintenance mode

Maintenance mode is a special configuration for running Kuma in read-only mode, where all operations that would write to the database are blocked. As the name suggests, it’s intended for those times when we’d like to continue to serve documents from a read-only copy of the database, while performing maintenance on the master database. For local Docker-based development in maintenance mode: 1. If you haven’t already, create a read-only user for your local MySQL database: docker-compose up -d docker-compose exec web mysql -h mysql -u root -p (when prompted for the password, enter "kuma") mysql> source ./scripts/create_read_only_user.sql mysql> quit

2. Create a .env file in the repository root directory, and add these settings: MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

Using a read-only database user is not required in maintenance mode. You can run in maintenance mode just fine with only this setting: MAINTENANCE_MODE=True

and going with a database user that has write privileges. The read-only database user simply provides a level of safety as well as notification (for example, an exception will be raised if an attempt to write the database slips through). 3. Update your local Docker instance: docker-compose up -d

4. You may need to recompile your static assets and then restart: docker-compose exec web make build-static docker-compose restart web

You should be good to go! There is a set of integration tests for maintenance mode. If you’d like to run them against your local Docker instance, first do the following: 1. Load the latest sample database (see Load the sample database). 2. Ensure that the test document “en-US/docs/User:anonymous:uitest” has been rendered (all of its macros have been executed). You can check this by browsing to http://localhost:8000/en-US/docs/User:anonymous:uitest. If there is no message about un-rendered content, you are good to go. If there is a message about un-rendered content, you will have to put your local Docker instance back into non-maintenance mode, and render the document:

2.2. Development 21 Kuma Documentation, Release latest

• Configure your .env file for non-maintenance mode: MAINTENANCE_MODE=False DATABASE_USER=root

• docker-compose up -d • Using your browser, do a shift-reload on http://localhost:8000/en-US/docs/User:anonymous:uitest and then put your local Docker instance back in maintenance mode: • Configure your .env file for maintenance mode: MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

• docker-compose up -d 3. Configure your environment with DEBUG=False because the maintenance-mode integration tests check for the non-debug version of the not-found page: DEBUG=False MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

This, in turn, will also require you to recompile your static assets: docker-compose up -d docker-compose exec web ./manage.py compilejsi18n docker-compose exec web ./manage.py collectstatic docker-compose restart web

Now you should be ready for a successful test run: py.test --maintenance-mode -m "not search" tests/functional --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver

Note that the “search” tests are excluded. This is because the tests marked “search” are not currently designed to run against the sample database.

2.2.15 Serving over SSL / HTTPS

Kuma can be served over HTTPS locally with a self-signed certificate. Browsers consider self-signed certificates to be unsafe, and you’ll have to confirm that you want an exception for this. 1. If you want GitHub logins: • In the Django Admin for Sites, ensure that site #2’s domain is set to developer.127.0.0.1.nip.io. • In GitHub, generate a new GitHub OAuth app for the test SSL domain, modifying the procees at Up- date the Sites section. When creating the GitHub OAuth app, replace http://localhost:8000 with https://developer.127.0.0.1.nip.io in both URLs. When creating the SocialApp in Kuma, chose the developer.127.0.0.1.nip.io site. 2. Include the SSL containers by updating .env: COMPOSE_FILE=docker-compose.yml:docker-compose.ssl.yml

3. Run the new containers:

22 Chapter 2. Getting started Kuma Documentation, Release latest

docker-compose up -d

4. Load https://developer.127.0.0.1.nip.io/en-US/ in your browser, and add an exception for the self-signed certifi- cate. 5. Load https://demos.developer.127.0.0.1.nip.io/en-US/ in your browser, and add an exception for the self-signed certificate again. Some features of SSL-protected sites may not be available, because the browser does not fully trust the self-signed SSL certificate. The HTTP-only website will still be available at http://localhost:8000/en-US/, but GitHub logins will not work.

2.2.16 Enabling PYTHONWARNINGS

Python ignores some warnings by default, including DeprecationWarning. To see these warnings, you can set the PYTHONWARNINGS environment variable in your .env file. For example: # Show every warning, every time it occurs PYTHONWARNINGS=always

Or alternatively: # Show every warning, but ignore repeats PYTHONWARNINGS=default

Note: Explicitly setting PYTHONWARNINGS=default will not do what you expect. It actually disables the default filters, ensuring that every warning gets displayed, but only the first time it occurs on a given line. See the PYTHONWARNINGS docs for more information on possible values.

2.2.17 Configuring AWS S3

The publish and unpublish Celery tasks and Django management commands require AWS S3 to be configured in order for them to do any real work, that is, creating/updating/deleting S3 objects used by the stage/production document API. In stage and production, the S3 bucket name as well as the AWS credentials are configured via the container environment, which in turn, gets the AWS credentials from a Kubernetes secrets resource. For local development, there is no need for any of this configuration. The publish and unpublish tasks will simply be skipped (although, for verification/debugging purposes, you can see the detailed skip messages in the worker log ( docker-compose logs -f worker). However, if for testing purposes you’d like to locally configure the publish and unpublish tasks to use S3, you can simply add the following to your .env file: MDN_API_S3_BUCKET_NAME= AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY=

2.2.18 Enabling django-querycount

If you want to find out how many SQL queries are made, per request, even if they are XHR requests, you can simply add this to your .env file: ENABLE_QUERYCOUNT=true

2.2. Development 23 Kuma Documentation, Release latest

Stop and start docker-compose and now, on stdout, it will print a table for every request URL about how many queries that involved and some information about how many of them were duplicates. If you want more insight into the duplicate queries add this to your .env: QUERYCOUNT_DISPLAY_DUPLICATES=3

A number greater than the (default) 0 means it will print the 3 most repeated SQL queries.

2.3 Troubleshooting

Kuma has many components. Even core developers need reminders of how to keep them all working together. This doc outlines some problems and potential solutions running Kuma.

2.3.1 Kuma “Reset”

These commands will reset your environment to a “fresh” version, with the current third-party libraries, while retaining the database: cd /path/to/kuma docker-compose down make clean git submodule sync --recursive && git submodule update --init --recursive docker-compose pull docker-compose build --pull docker-compose up

2.3.2 Reset a corrupt database

The Kuma database can become corrupted if the system runs out of disk space, or is unexpectedly shutdown. MySQL can repair some issues, but sometimes you have to start from scratch. When this happens, pass an extra argument --volumes to down: cd /path/to/kuma docker-compose down --volumes make clean git submodule sync --recursive && git submodule update --init --recursive docker-compose pull docker-compose build --pull docker-compose up -d mysql sleep 20 # Wait for MySQL to initialize. See notes below docker-compose up

The --volumes flag will remove the named MySQL database volume, which will be recreated when you run docker-compose up. The mysql container will take longer to start up as it recreates an empty database, and the kuma container will fail until the mysql container is ready for connections. The 20 second sleep should be sufficient, but if it is not, you may need to cancel and run docker-compose up again. Once mysql is ready for connections, follow the installation instructions, starting at Provisioning a database, to configure your now empty database.

24 Chapter 2. Getting started Kuma Documentation, Release latest

2.3.3 Run alternate services

Docker services run as containers. To change the commands or environments of services, it is easiest to add an override configuration file, as documented in Customizing the Docker environment.

2.3.4 Linux file permissions

On Linux, it is common that files created inside a Docker container are owned by the root user on the host system. This can cause problems when trying to work with them after creation. We are investigating solutions to create files as the developer’s user. In some cases, you can specify the host user ID when running commands: docker-compose run --rm --user $(id -u) web ./manage.py collectstatic

In other cases, the command requires root permissions inside the container, and this trick can’t be used. Another option is to allow the files to be created as root, and then change them to your user on the host system: find . -user root -exec sudo chown $(id -u):$(id -g) \{\} \;

2.3.5 KumaScript macros are not evaluating

If you’re seeing tags like {{HTMLSidebar}} or {{HTMLElement("head")}} it could be happening because there is an outdated macro that needs to be removed. For example on /en-US/docs/Web/HTML, there is a deleted macro called CommunityBox. To fix this, log in to edit the page, remove the CommunityBox macro, then click “Publish”. Visit the affected page again and you should see actual content instead of the macros. Note: Sometimes the wiki site (e.g. http://wiki.localhost.org:8000/en-US/docs/Web/HTML$edit) will throw an error after editing the page, acting as if it didn’t save your edit. View the actual URL of the page (e.g. http://localhost.org:8000/en-US/docs/Web/HTML) to verify that the changes were accepted.

2.3.6 Fixing a 404 error on a specific page

This most likely happens because the document isn’t in the database yet, so it needs to be scraped. Run the command ./manage.py scrape_document [url of document], e.g.: docker-compose exec web ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Glossary/HTML

Memorize or have handy the ./manage.py scrape_document command since this will come up often. See Add an MDN Wiki Document for more details.

2.3.7 Getting more help

Check if there is anything helpful in the logs: docker-compose logs web kumascript docker-compose logs web

If you have more problems running Kuma, please: 1. Paste errors to pastebin.

2.3. Troubleshooting 25 Kuma Documentation, Release latest

2. Start a thread on Discourse. 3. After you email dev-mdn, you can also ask in the #mdn-dev Slack channel.

2.4 The Kuma test suite

Kuma has a fairly comprehensive Python test suite. Changes should not break tests. Only change a test if there is a good reason to change the expected behavior. New code should come with tests. Commands should be run inside the development environment, after make bash.

2.4.1 Setup

Before you run the tests, you have to build assets: make build-static

2.4.2 Running the test suite

If you followed the steps in the installation docs, then all you should need to do to run the test suite is: make test

The default options for running the test are in pytest.ini. This is a good set of defaults. If you ever need to change the defaults, you can do so at the command line by running what the Make task does behind the scenes: py.test kuma

Some helpful command line arguments to py.test (won’t work on make test): --pdb: Drop into pdb on test failure. --create-db: Create a new test database. --showlocals: Shows local variables in tracebacks on errors. --exitfirst: Exits on the first failure. See py.test --help for more arguments.

Running subsets of tests and specific tests

There are a bunch of ways to specify a subset of tests to run: • only tests marked with the ‘spam’ marker: py.test -m spam

• all the tests but those marked with the ‘spam’ marker: py.test -m "not spam"

• all the tests but the ones in kuma/core:

26 Chapter 2. Getting started Kuma Documentation, Release latest

py.test --ignore kuma/core

• all the tests that have “foobar” in their names: py.test -k foobar

• all the tests that don’t have “foobar” in their names: py.test -k "not foobar"

• tests in a certain directory: py.test kuma/wiki/

• specific test: py.test kuma/wiki/tests/test_views.py::RedirectTests::test_redirects_only_internal

See http://pytest.org/latest/usage.html for more examples.

Showing test coverage

While running the tests you can record which part of the code base is covered by test cases. To show the results at the end of the test run use this command: make coveragetest

To generate an HTML coverage report, use: make coveragetesthtml

The test database

The test suite will create a new database named test_%s where %s is whatever value you have for settings.DATABASES[’default’][’NAME’]. Make sure the user has ALL on the test database as well.

2.4.3 Markers

See: py.test--markers

for the list of available markers. To add a marker, add it to the pytest.ini file. To use a marker, add a decorator to the class or function. Examples: import pytest

@pytest.mark.spam class SpamTests(TestCase): ...

class OtherSpamTests(TestCase): @pytest.mark.spam def test_something(self): ...

2.4. The Kuma test suite 27 Kuma Documentation, Release latest

2.4.4 Adding tests

Code should be written so that it can be tested, and then there should be tests for it. When adding code to an app, tests should be added in that app that cover the new functionality. All apps have a tests module where tests should go. They will be discovered automatically by the test runner as long as the look like a test.

2.4.5 Changing tests

Unless the current behavior, and thus the test that verifies that behavior is correct, is demonstrably wrong, don’t change tests. Tests may be refactored as long as it’s clear that the result is the same.

2.4.6 Removing tests

On those rare, wonderful occasions when we get to remove code, we should remove the tests for it, as well. If we liberate some functionality into a new package, the tests for that functionality should move to that package, too.

2.5 Client-side testing

Kuma has a suite of functional tests using pytest. All these test suites live in the /tests/ directory. The tests directory comprises of: • /headless contains tests that don’t require a browser • /utils contains helper functions.

2.5.1 Setting up test environment

1. In the terminal go to the directory where you have downloaded kuma. 2. Install the requirements the tests need to run: $ poetry install

2.5.2 Running the tests

The minimum amount of information necessary to run the tests against the staging server is the directory the tests are in. Currently, all of the tests should run successfully against the staging server.: $ poetry run pytest tests

This basic command can be modified to run the tests against a local environment, or to run tests concurrently.

Only running tests in one file

Add the name of the file to the test location: $ poetry run pytest tests/headless/test_cdn.py

28 Chapter 2. Getting started Kuma Documentation, Release latest

Run the tests against a different url

By default the tests will run against the staging server. If you’d like to run the tests against a different URL (e.g., your local environment) pass the desired URL to the command with --base-url: $ poetry run pytest tests --base-url http://mdn.localhost:8000

Run the tests in parallel

By default the tests will run one after the other but you can run several at the same time by specifying a value for -n: $ poetry run pytest tests -n auto

2.5.3 Using Alternate Environments

Run tests on MDN’s Continuous Integration (CI) infrastructure

If you have commit rights on the mdn/kuma GitHub repository you can run the functional tests using the MDN CI Infrastructure. Just force push to mdn/kuma@stage-integration-tests to run the tests against https://developer.allizom.org. You can check the status, progress, and logs of the test runs at MDN’s Jenkins-based multi-branch pipeline.

2.5.4 MDN CI Infrastructure

The MDN CI infrastructure is a Jenkins-based, multi-branch pipeline. The pipelines for all branches are defined by the Jenkinsfile and the files under the Jenkinsfiles directory. The basic idea is that every branch may have its own custom pipeline steps and configuration. Jenkins will auto-discover the steps and configuration by checking within the Jenkinsfiles directory for a Groovy (.groovy) and/or YAML (.yml) file with the same name as the branch. For example, the “stage-integration-tests” branch has a Jenkinsfiles/stage-integration-tests.yml file which will be loaded as configuration and used to determine what to do next (load and run the Groovy script specified by its pipeline.script setting - Jenkinsfiles/integration- tests.groovy - and the script, in turn, will use the dictionary of values provided by the job setting defined within the configuration). Note that the YAML files for the integration-test branches provide settings for configuring things like the Dockerfile to use to build the container, and the URL to test against. The integration-test Groovy files use a number of global Jenkins functions that were developed to make the build- ing, running and pushing of Docker containers seamless (as well as other cool stuff, see mozmar/jenkins-pipeline). They allow us to better handle situations that have been painful in the past, like the stopping of background Docker containers. The “prod-integration-tests” branch also has its own Jenkinsfiles/prod-integration-tests.yml file. It’s identical to the YAML file for the “stage-integration-tests” branch except that it specifies that the tests should be run against the production server rather than the staging server (via its job.base_url setting). Similarly, the “master” branch has it’s own pipeline, but instead of being configured by a YAML file, the entire pipeline is defined within its Jenkinsfiles/master.groovy file. The pipeline for any other branch which does not provide its own Groovy and/or YAML file will follow that defined by the Jenkinsfiles/default.groovy file. You can check the status, progress, and logs of any pipeline runs via MDN’s Jenkins-based multi-branch pipeline.

2.5. Client-side testing 29 Kuma Documentation, Release latest

2.5.5 Markers

• nondestructive Tests are considered destructive unless otherwise indicated. Tests that create, modify, or delete data are consid- ered destructive and should not be run in production. • smoke These tests should be the critical baseline functional tests.

2.5.6 Guidelines for writing tests

See Bedrock and the Web QA Style Guide.

2.6 Feature toggles

MDN uses feature toggles to integrate un-finished feature changes as early as possible, and to control the behavior of finished features. Some site features are controlled using django-waffle. You control these features in the django admin site’s waffle section. Some site features are controlled using constance. You control these features in the django admin site’s constance section.

2.6.1 Waffle features

Switches

Waffle switches are simple booleans - they are either on or off. • application_ACAO - Enable Access-Control-Allow-Origin=0 header. • foundation_callout - Show foundation donate homepage tile • helpful-survey-2 - Enable 2017 Helpfulness Survey • registration_disabled - Enable/disable new user registration. • store_revision_ips - Save request data, including the IP address, to enable marking revisions as spam. • welcome_email - Send welcome email to new user registrations. • wiki_spam_training - Call Akismet to check submissions, but don’t block due to detected spam or Ak- ismet errors. • developer_needs - Enable/disable the developer needs survey banner

Flags

Waffle flags control behavior by specific users, groups, percentages, and other advanced criteria. • contrib_beta - Enable/disable the contributions popup and pages • kumaediting - Enable/disable wiki editing. • page_move - (deprecated) enable/disable page move feature.

30 Chapter 2. Getting started Kuma Documentation, Release latest

• section_edit - Show section edit buttons. • sg_task_completion - Enable the Survey Gizmo pop-up. • spam_admin_override - Tell Akismet that edits are never spam. • spam_checks_enabled - Toggle spam checks site wide. • spam_spammer_override - Tell Akismet that edits are always spam. • spam_submissions_enabled - Toggle Akismet spam/spam submission ability. • spam_testing_mode - Tell Akismet that edits are tests, not real content. • subscription_banner - Enable/disable the subscriptions banner • subscription_form - Enable/disable the subscriptions form

2.6.2 Constance features

Constance configs let us set operational values for certain features in the database - so we can control them without changing code. They are all listed and documented in the admin site’s constance section.

2.6.3 Using Traffic Cop for A/B Testing

Traffic Cop is a lightweight JavaScript library that allows conducting content experiments on MDN without the need for third party tools such as Optimizely. Traffic Cop also respects user privacy, and will not run if Do Not Track is enabled. On MDN, Traffic Cop will also not run for logged in users. Here’s how to implement a Traffic Cop content experiment. The first task is to choose your control page and create your tests pages Here we will use the scenario of testing a new feature on MDN. The new feature involved replacing the static examples on top, with an interactive editor. First we choose our control page. For this example, we will use: https://developer.mozilla.org/docs/Experiment:ExperimentName/Array.prototype.push() Next we need our test page. The convention on MDN is to first create a base page that follows a naming convention such as: https://developer.mozilla.org/docs/Experiment:ExperimentName/ With the base page created, we next create out test variation of our chosen page. To do this, navigate to: https://developer.mozilla.org/docs/Experiment:ExperimentName/Array.prototype.push() You should should now be presented with an editing interface to create the new page. Switch back to the control page, and click the Edit button. Switch the editor to source mode and copy all of the contents inside the editor. Switch back to our test, ensure the editor is set to source mode, and paste the content you just copied. You can now go ahead and make the changes you wish to user test. Once you are complete, go ahead and publish the page.

The JavaScript

Let us first add the JavaScript code to initialise and configure Traffic Cop. Inside kuma\static\js, create a new file called some like experiment-experiment-name.js and paste the following code into the file:

2.6. Feature toggles 31 Kuma Documentation, Release latest

1 (function() {

2 'use strict';

3

4 var cop= new Mozilla.TrafficCop({

5 id: 'experiment-experiment-name',

6 variations:{

7 'v=a': 50, // control

8 'v=b': 50 // test

9 }

10 });

11

12 cop.init();

13 })(window.Mozilla);

This will initialise Traffic Cop, set up an experiment with the id experiment-experiment-name, and lastly define a 50/50 split between the control, and the test page.

Define your bundle

Next we need to add an entry into kuma\settings\common.py to identify the test, and the related JS that will be injected. Find the following line in common.py: PIPELINE_JS = {

Just before the closing } add a block such as the following:

1 'experiment-experiment-name':{

2 'source_filenames':(

3 'js/libs/mozilla.dnthelper.js',

4 'js/libs/mozilla.cookiehelper.js',

5 'js/libs/mozilla.trafficcop.js',

6 'js/experiment-experiment-name.js',

7 ),

8 'output_filename':'build/js/experiment-experiment-name.js',

9 },

NOTE: The key here experiment-experiment-name needs to match the id you specified in your JS file above.

Identify your A/B test pages

The final step is to identify the pages to the back-end so it will know where to direct traffic based on the URL parameter that will be added by Traffic Cop. Inside kuma\settings\content_experiments.json add the following:

1 [

2 {

3 "id": "experiment-experiment-name",

4 "ga_name": "experiment-name",

5 "param": "v",

6 "pages":{

7 "en-US:Web/JavaScript/Reference/Global_Objects/Array/push":{

8 "a": "Web/JavaScript/Reference/Global_Objects/Array/push",

9 "b": "Experiment:ExperimentName/Array.prototype.push()"

10 },

11 }

12 }

13 ]

32 Chapter 2. Getting started Kuma Documentation, Release latest

There are a couple of important points to note here: 1. We are leaving of the domain, as well as the docs/ part of the url. 2. As with the entry in common.py, the id here matches the id in your JS file, tying it all together. 3. The param value, needs to match the string you specified inside the variations block in your JS 4. The first part of our key under pages above, identifies the locale to which this will apply, en-US in this case. 5. The key for the two pages listed next, needs to match the values you used as the parameter value inside variations in your JS file earlier.

Testing your experiment

With you local instance of Kuma running, navigate to the page you defined as your control. In this example: http://localhost:8000/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/push NOTE: You should not be logged in to MDN, and ensure that Do Not Track is disabled. Your experiment JavaScript code bundle defined in common.py should be injected into the page, and Traffic Cop will add a URL parameter to the page that is either v=a or v=b. Depending on which, you will either see the control(a), or the variation(b). You can also force a specific page to load by appending ?v=a or, ?v=a manually to the control page URL. If all the above works as expected, open up a pull request, and tag someone on MDN for reivew.

2.7 Celery and async tasks

Kuma uses Celery to enable asynchronous task processing for long-running jobs and to speed up the request-response cycle.

2.7.1 When is Celery appropriate

You can use Celery to do any processing that doesn’t need to happen in the current request-response cycle. Ask yourself the question: “Is the user going to need this data on the page I’m about to send them?” If not, using a Celery task may be a good choice. Some examples where Celery is used: • Sending notification emails on page changes - If this was done in the page change request, then pages with many watchers would be slower to edit, and errors in email sending would result in a “500 Internal Server Error” for the editor. • Populating the contributor bar - Gathering and sorting the contributor list can be a slow process that would delay rendering of stale pages. Instead, viewers quickly see the content, possibly without the contributor bar. Once the async task has populated the cache, future viewers get the content and the contributor bar. • Generating the spam moderator dashboard - Calculating spam statistics is a potentially long process, and if it takes more than 30 seconds the viewer will get a “502 Bad Gateway” error. The async task can take as long as needed, and the spam moderator will see the dashboard when the data is calculated and cached. • Rebuilding sitemaps - Search engines need recent site URL data, and it can’t be assembled quickly in the request for the sitemap. An async request repopulates the cached sitemap data, keeping this data fast and up-to- date. There are some downsides to async tasks that should also be considered:

2.7. Celery and async tasks 33 Kuma Documentation, Release latest

• Message passing, serialization, de-serialization, and data reloading increase the load on the entire system. • Async tasks require different testing strategies. It is easy to write a passing test that doesn’t reflect how the task is called in production. • Runtime errors are more difficult to troubleshoot with async tasks, due to missing context on how they were called. In general, it is better to get an algorithm right in the request loop, and only move it to an asynchronous task when it is identified as a performance issue.

2.7.2 Celery services

A working Celery installation requires several services.

Worker

Celery processes tasks with one or more workers. In Kuma, the workers and web processes share a code base, so that Django models, functions, and settings are available to async tasks, and web code can easily schedule async tasks. In Docker, the worker process runs in the worker service / container.

Broker

Kuma uses Redis as the message broker. There are some caveats to be aware of. Redis is used both in local develop- ment and in production.

Result store

When a task completes, it returns processed data and task states to a results store. Kuma doesn’t use returned data, but it does use returned task state to coordinate multi-step tasks. In particular for chord tasks such as building a sitemap of sitemaps after each locales’ sitemap is built. Referring to logging to see message about completion of Celery worker tasks.

Periodic tasks scheduler

Periodic tasks are scheduled with celery beat, which adds tasks to the task queue when they become due. It should only be run once in a deployment, or tasks may be scheduled multiple times. In Docker, it runs in the worker container by starting the celery process with --beat. In production, there are several task workers, and the celery beat process is run directly on just one worker. All scheduled periodic tasks are configured in code. As a pattern, each Django app (e.g. kuma.wiki) that has tasks to run periodically are wired up within the ready method of each app’s app config class (e.g. kuma.wiki.apps.WikiConfig).

2.7.3 Configuring and running Celery

We set some reasonable defaults for Celery in kuma/settings/common.py. These can be overridden by the environment variables, including:

34 Chapter 2. Getting started Kuma Documentation, Release latest

• CELERY_TASK_ALWAYS_EAGER Default: False (Docker), True (tests). When True, tasks are executed immediately, instead of being scheduled and executed by a worker, skipping the broker, results store, etc. In theory, tasks should act the same whether executed eagerly or normally. In practice, there are some tasks that fail or have different results in the two modes, mostly due to database transactions. • CELERY_WORKER_CONCURRENCY Default: 4. Each worker will execute this number of tasks at the same time. 1 is a good value for debugging, and the number of CPUs in your environment is good for performance. The worker can also be adjusted at the command line. For example, this could run inside the worker service con- tainer: celery -A kuma.celery:app worker --log-level=DEBUG -c 10

This would start Celery with 10 worker threads and a log level of DEBUG.

2.8 Search

Kuma uses Elasticsearch to power its on-site search.

2.8.1 Installed version

Elasticsearch releases new versions often. They release a new minor version (for example, 6.2.0, 6.3.0) every 30 to 90 days, and support the most recent minor version. They release a major version (5.0.0, 6.0.0) every 12 to 18 months, and support the last minor version of the previous major release (for example, 2.4.x supported until 6.x is released, 5.6.x supported until 7.x is released). Kuma’s goal is to update to the last minor version of the previous major release, so we expect to update every 12 to 18 months. Kuma is running on ElasticSearch 6.7.1 in development and production, which is planned to be supported until September 2020. The Kuma search tests use the configured Elasticsearch, and can be run inside the web Docker container to confirm the engine works: py.test kuma/search/

2.8.2 Indexing documents

To see search results, you will need to index the documents.

Using the Admin

This process works both in a development environment and in production: • Open the Django admin search index list view. On Docker, this is located at http://localhost.org:8000/admin/search/index/. • On the search index list, select the checkbox next to the each index. Then select “Delete selected Indexes” from the dropdown menu.

2.8. Search 35 Kuma Documentation, Release latest

• Inside the web Docker container: docker-compose exec web ./manage.py reindex

• Refresh the search index list until the “populated” field changes to a green checkbox image. In production, you will also get an email notifying you when the index is populated. • Select the checkbox next to the populated index, then choose “Promote selected search index to current index” in the actions dropdox. Click “Go” to promote the index. • All three fields will be green checkboxes (promoted, populates, and “is current index?”). The index is live, and will be used for site search. Similarly you can also demote a search index and it will automatically fall back to the previously created index (by created date). That helps to figure out issues in production and should allow for a smooth deployment of search index changes. It’s recommended to keep a few search indexes around just in case. If no index is created in the admin UI the fallback “main_index” index will be used instead. When you delete a search index in the admin interface, it is deleted on Elasticsearch as well.

Using the shell

Inside the web Docker container: ./manage.py reindex

This will populate and activate the fallback index “main_index”. It will be overwritten when the search tests run.

2.8.3 Search Filters

The search interface uses Filters and Filter Groups to determine what results are shown. Currently, the only active Filter Group is “Topics”. The Filters collect documents by tag. For example, the Filter “Add-ons & Extensions” collects all documents tagged with “Add-ons”, “Extensions”, “Plugins”, or “Themes”. Most filters have a single associated tag. For example, the “CSS” filter collects the documents with the “CSS” tag. Filter and Filter Groups are defined in the database, but the names are shown to users. The names of these objects (enabled and disabled) are listed in kuma/search/names.py, and marked for translation. To generate this file, run this command against the production database, and copy the output to names.py: ./manage.py generate_search_names

The sample database contains the active subset of Filters and Filter Groups. This won’t match the production database, and so the development environment should not be used to generate kuma/search/names.py.

2.9 Localization

Kuma is localized with gettext. User-facing strings in the code or templates need to be marked for gettext localization. We use Pontoon to provide an easy interface to localizing these files. Pontoon allows translators to use the web UI as well as download the PO files, use whatever tool the translator feels comfortable with and upload it back to Pontoon. The strings are stored in a separate repository, https://github.com/mozilla-l10n/mdn-l10n

Note: We do not accept pull requests for updating translated strings. Please use Pontoon instead.

36 Chapter 2. Getting started Kuma Documentation, Release latest

See the Django documentation on Translations for how to make strings marked for translation in Python and templates. Unless otherwise noted, run all the commands in this document inside the development environment. For Docker on Linux, set UID in .env or enter the environment with docker-compose run --rm --user $(id -u) web bash, to ensure that created files are owned by your development user.

2.9.1 Build the localizations

Localizations are found in this repository under the locale folder. This folder is a git submodule, linked to the mdn-l10n repository. The gettext portable object (.po) files need to be compiled into the gettext machine object (.mo) files before trans- lations will appear. This is done once during initial setup and provisioning, but will be out of date when the kuma locales are updated. To refresh the translations, first update the submodule in your host system: git submodule update --init --depth=10 locale

Next, enter the development environment, then: 1. Compile the .po files: make localecompile

2. Update the static JavaScript translation catalogs: make compilejsi18n

3. Collect the built files so they are served with requests: make collectstatic

2.9.2 Update the localizations in Kuma

To get the latest localization from Pontoon users, you need to update the submodule.

Note: This task is done by MDN staff or by automated tools during the push to production. You should not do this as part of a code-based Pull Request.

On the host system: 1. Create a new branch from kuma master:

git remote -v | grep origin # Should be main kuma repo git fetch origin git checkout origin/master git checkout -b update-locales # Pick your own name. Can be combined # with updating the kumascript submodule.

2. Update the locale submodule to master: cd locale git fetch git checkout origin/master cd ..

2.9. Localization 37 Kuma Documentation, Release latest

3. Commit the update: git commit locale -m "Updating localizations"

4. Push to GitHub (your fork or main repository), and open a Pull Request. It is possible to break deployments by adding a bad translation. The TravisCI job TOXENV=locales will test that the deployment should pass, and should pass before merging the PR.

2.9.3 Update the localizable strings in Pontoon

When localizable strings are added, changed, or removed in the code, they need to be gathered into .po files for trans- lation. The TravisCI job TOXENV=locales attempts to detect when strings change by displaying the differences in locale/templates/LC_MESSAGES/django.pot, but only when a msgid changes. When this happens, the strings need to be exported to the mdn-l10n repository so that they are available in Pontoon. If done incorrectly, then the work of localizers can be lost. This task is done by staff when preparing for deployment.

2.9.4 Add a new locale to Pontoon

The process for getting a new locale on MDN is documented at Starting a new MDN localization. One step is to enable translation of the UI strings. This will also enable the locale in development environments and on https://developer.allizom.org.

Note: This task is done by MDN staff.

This example shows adding a Bulgarian (bg) locale. Change bg to the locale code of the language you are adding. 1. Updating the localizable strings in Pontoon as above, so that your commit will be limited to the new locale. 2. In kuma/settings/common.py, add the locale to ACCEPTED_LOCALES and CANDIDATE_LOCALES, and increase PUENTE[’VERSION’]. 3. Download the latest languages.json from https://product-details.mozilla.org/1.0/languages.json and place it at kuma/settings/languages.json. 4. Add the locale to translate_locales.html and the locale/ folder: make locale LOCALE=bg

5. Generate the compiled files for all the locales, including the new one: make localerefresh

6. Restart the web server and verify that Django loads the new locale without errors by visiting the locale’s home page, for example http://localhost:8000/bg/. 7. Commit the locale submodule and push to mdn-l10n, as described above in Updating the localizable strings in Pontoon. The other locales should include a new string representing the new language. 8. (Optional) Generate migrations that includes the new locale: ./manage.py makemigrations users wiki --name update_locale

9. Commit the changes to locale, jinja2/includes/translate_locales.html, and kuma/settings, and open a Pull Request. 10. Enable the language in Pontoon, and notify the language community to start UI translations.

38 Chapter 2. Getting started Kuma Documentation, Release latest

2.9.5 Enable a new locale on MDN

Once the new translation community has completed the rest of the process for starting a new MDN localization, it is time to enable the language for page translations:

Note: This task is done by MDN staff.

1. Remove the locale from CANDIDATE_LOCALES in kuma/settings/common.py. Ensure it remains in ACCEPTED_LOCALES. 2. Restart the web server and verify that Django loads the new locale without errors by visiting the locale’s home page, for example http://localhost:8000/bg/. 3. Commit the change to kuma/settings/common.py and open a Pull Request. When the change is merged and deployed, inform the localization lead and the community that they can begin trans- lating content.

2.10 CKEditor

MDN Web Docs uses a WYSIWYG editor called CKEditor. CKEditor is an open source utility which brings the power of rich text editing to the web. This document details how to update CKEditor within the MDN codebase.

2.10.1 Building CKEditor

To rebuild CKEditor, run this on the host system: cd assets/ckeditor4/ ./docker-build.sh # Creates a Java container, builds CKEditor docker-compose exec web make build-static

This builds CKEditor within a Docker VM, using the Java ckbuilder tool from CKSource, then builds static resources so that the updated editor is installed where it belongs. To rebuild CKEditor locally, if you have Java installed: cd assets/ckeditor4/ ./build.sh docker-compose exec web make build-static

Portions of the build process will take a few minutes so don’t expect an immediate result.

2.10.2 Updating CKEditor

To update the CKEditor version, you’ll need to edit build.sh and change the value of CKEDITOR_VERSION. Updating is important to keep MDN in sync with CKEditor’s functional and security updates.

2.10.3 Updating CKEditor plugins

Some plugins are maintained by MDN staff in the Kuma repo. Others are updated to the tag or commit number specified in build.sh: • descriptionlist

2.10. CKEditor 39 Kuma Documentation, Release latest

• scayt • wsc • wordcount Change to CKEditor plugins which are bundled into Kuma should be made in the directory /as- sets/ckeditor4/source/plugins/. Once you’ve made changes to a plugin, be sure to build the editor and the static resources again, as described in Building CKEditor.

2.10.4 Interactively developing CKEditor plugins

For small changes, it may be fastest to rebuild CKEditor for each change. For more involved changes, you can load CKEditor in development mode. To interactively develop CKEditor plugins, first download the CKEditor source code on the host system: cd assets/ckeditor4/source/ ./build.sh --download

Enable development mode by adding a line to .env: CKEDITOR_DEV=True

Restart Docker with docker-compose restart web to apply the new setting. This will switch to a mode where plugins are loaded dynamically, based on the list in /kuma/wiki/jinja2/wiki/includes/ckeditor_scripts.html, and you can see your changes by reloading the browser. When you are done, disable development mode in .env: CKEDITOR_DEV=False # Or comment out the line

Restart Docker with docker-compose restart web, and follow the instructions above for Building CKEditor, to delete the CKEditor source code and rebuild CKEditor with your changes.

2.10.5 Committing Changes

When updating CKEditor, adding a plugin, or changing the configuration, CKEditor should be rebuilt, and the results added to a new pull request. We currently do not check-in CKEditor source, but do add third-party plugin sources. We check in the “built” files, which combine CKEditor with plugins and configuration, so that they do not need to be rebuilt by the static files process. This is enforced by .gitignore, so git add can be used: git add assets/ckeditor4

Reviewers should rebuild CKEditor to ensure this was done correctly. Building is not atomic, because a hex-encoded timestamp is embedded in some minified files: assets/ckeditor4/build/ckeditor/ckeditor.js assets/ckeditor4/build/ckeditor/skins/moono/editor.css assets/ckeditor4/build/ckeditor/skins/moono/editor_*.css

Small changes (with big diffs) to these files are expected. Changes to other files are not expected.

40 Chapter 2. Getting started Kuma Documentation, Release latest

2.11 Getting Data

A Kuma instance without data is useless. You can view the front page, but almost all interactive testing requires users, wiki documents, and other data.

2.11.1 The Sample Database

The sample database has a minimal set of data, based on production data, that is useful for manual and automated testing. This includes: • All documents linked from the English homepage, including: – Translations – Some historical revisions – Minimal author profiles for revisions • Additional documents for automated testing and feature coverage • Good defaults for contance variables KUMASCRIPT_TIMEOUT and KUMASCRIPT_MAX_AGE • Waffle flags and switches • Search filters and tags (but not a search index, which must be created locally - see Indexing documents) • The Mozilla Hacks feed • Test users, with appropriate groups and permissions: – test-super - A user with full permissions – test-moderator - A staff content moderator – test-new - A regular user account – test-banned - A banned user – viagra-test-123 - An unbanned spammer Test accounts can be accessed by entering the password test-password in the Django admin, and then returning to the site. See Load the sample database for instructions on loading the latest sample database.

2.11.2 Add an MDN User

If you need a user profile that is not in the sample database, you can scrape it from production or another Kuma instance, using the scrape_user command. In the container (after make bash or similar), run the following, replacing username with the desired user’s username: ./manage.py scrape_user username ./manage.py scrape_user https://developer.mozilla.org/en-US/profiles/username

Some useful options: --email [email protected] Set the email for the user, which can’t be scraped from the profile. With the correct email, the user’s profile image will be available. --social Scrape social data for the user, which is not scraped by default --force If a user exists in the current database, update it with scraped data.

2.11. Getting Data 41 Kuma Documentation, Release latest

For full options, see ./manage.py scrape_user --help A local user can be promoted to staff with the command: ./manage.py ihavepower username --password=password

2.11.3 Add an MDN Wiki Document

If you need a wiki page that is not in the sample database, you can scrape it from production or another Kuma instance, using the scrape_document command. In the container (after make bash or similar), run the following, using the desired URL: ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS/display

You can also pass in multiple URLs instead of doing one URL at a time. Scraping a document includes: • The parent documents (such as Web and Web/CSS for Web/CSS/display) • A revision for each document, and the author for each revision • The English variant, if a translation is requested • Redirects, if a page move is detected It does not include: • Attachment data, or the attachments themselves. These will continue to use production URLs. Some useful options: --revisions REVS Scrape more than one revision --translations Scrape the translations of a page as well --depth DEPTH Scrape one or more levels of child pages as well --force Refresh an existing Document, instead of skipping For full options, see ./manage.py scrape_document --help

2.11.4 Add Documents Linked from a Page

If you need all the documents linked from a page in production or another Kuma instance, you can use the scrape_links command. In the container (after make bash or similar), run the following, using the desired URL: ./manage.py scrape_links # Scrape the homepage ./manage.py scrape_links https://developer.mozilla.org/en-US/Web/CSS/display

This treats a page a lot like a web crawler would, looking for wiki document links with the same locale from: • The header • The footer • The content • KumaScript-rendered sidebars and content This can result in a lot of traffic. There are options that don’t affect the initial link scrape, but that are passed on to the scraped documents:

42 Chapter 2. Getting started Kuma Documentation, Release latest

--revisions REVS Scrape more than one revision --translations Scrape the translations of a page as well --depth DEPTH Scrape one or more levels of child pages as well

2.11.5 Create the Sample Database

These scraping tools are used to create a sample database of public information, which is used for development environments and functional testing without exposing any private production data. When it is time to create a new sample database, an MDN staff person runs the commamd in the the container: time scripts/create_sample_db.sh

This takes 2 to 2½ hours with a good internet connection. This is then uploaded to the mdn-downloads site: • https://mdn-downloads.s3-us-west-2.amazonaws.com/index.html • https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz This uses the specification at etc/sample_db.json, which includes the sources for scraping, as well as fixtures needed for a working development and testing environment.

2.11.6 Load Custom Data

The sample_mdn command does the work of creating the sample database. It can also be used with a different specification to load custom fixtures and scrape additional data for your local environment. For example, loading a new sample database wipes out existing data, so you’ll need to run the instructions in Update the Sites section again. Instead, you can create a specification for your development user and GitHub OAuth application: { "sources":[ ["user", "my_username", { "social": true, "email":"[email protected]" } ] ], "fixtures":{ "users.user":[ { "username":"my_username", "email":"[email protected]", "is_staff": true, "is_superuser": true } ], "socialaccount.socialapp":[ { "name":"GitHub", "client_id":"client_id_from_github", "secret":"secret_from_github", "provider":"github", "sites":[[1]] }

2.11. Getting Data 43 Kuma Documentation, Release latest

], "socialaccount.socialaccount":[ { "uid":"uid_from_github", "user":["my_username"], "provider":"github" } ], "account.emailaddress":[ { "user":["my_username"], "email":"[email protected]", "verified": true } ] } }

To use this, you’ll need to replace the placeholders: • my_username - your MDN username • [email protected] - your email address, verified on GitHub • client_id_from_github - from your GitHub OAuth app • secret_from_github - from your GitHub OAuth app • uid_from_github - from your MDN SocialAccount Save it, for example as my_data.json, and, after loading the sample database, load the extra data: ./manage.py sample_mdn my_data.json

This will allow you to quickly log in again using GitHub auth after loading the sample database.

2.11.7 Anonymized Production Data

The production database contains confidential user information, such as email addresses and authentication tokens, and it is not distributed. We try to make the sample database small but useful, and provide scripts to augment it for specific uses, reducing the need for production data. Production-scale data is occasionaly needed for development, such as testing the performance of data migrations and new algorithms, and for the staging site. In these cases, we generate an anonymized copy of production data, which deletes authentication keys and anonymizes user records. This is generated with the script scripts/clone_db.py on a recent backup of the production database. You can see a faster and less resource-intensive version of the process by running it against the sample database: scripts/clone_db.py -H mysql -u root -p kuma -i mdn_sample_db.sql.gz anon_db

This will generate a file anon_db-anon-20170606.sql.gz, where the date is today’s date. To check that the anonymize script ran correctly, load the anonymized database dump and run the check script: zcat anon_db-anon-20170606.sql.gz | ./manage.py dbshell cat scripts/check_anonymize.sql | ./manage.py dbshell

This runs a set of counting queries that should return 0 rows. A similar process is used to anonymize a recent production database dump. The development environment is not tuned for the I/O, memory, and disk requirements, and will fail with an error. Instead, a host-installed version of MySQL

44 Chapter 2. Getting started Kuma Documentation, Release latest is used, with the custom collation. The entire process, from getting a backup to uploading a confirmed anonymized database, takes about half a day. We suspect that a clever user could de-anonymize the data in the full anonymized database, so we do not distribute it, and try to limit our own use of the database.

2.12 Documentation

This documentation is generated and published at Read the Docs whenever the master branch is updated. GitHub can render our .rst documents as ReStructuredText, which is close enough to Sphinx for most code reviews, without features like links between documents. It is occasionally necessary to generate the documentation locally. It is easiest to do this with a virtualenv on the host system, using Docker only to regenerate the MDN Sphinx template. If you are not comfortable with that style of development, it can be done entirely in Docker using docker-compose.

2.12.1 Generating documentation

Sphinx uses a Makefile in the docs subfolder to build documentation in several formats. MDN only uses the HTML format, and the generated document index is at docs/_build/html/index.html. To generate the documentation in a virtualenv on the host machine, first install the requirements: pip install -r docs/requirements.txt

Then switch to the docs folder to use the Makefile: cd docs make html python -m webbrowser file://${PWD}/_build/html/index.html

To generate the documentation with Docker: docker-compose run --rm web sh -c "\ python -m venv /tmp/.venvs/docs && \ . /tmp/.venvs/docs/bin/activate && \ pip install -r /app/docs/requirements.txt && \ cd /app/docs && \ make html" python -m webbrowser file://${PWD}/docs/_build/html/index.html

A virtualenv is required, to avoid a pip bug when changing the version of a system-installed package.

2.13 Docker

Docker is used for development and for deployment.

2.13.1 Docker Images

Docker images are used in development, usually with the local working files mounted in the images to set behaviour.

2.12. Documentation 45 Kuma Documentation, Release latest

Images are built by Jenkins, after tests pass, and are published to DockerHub. We try to store the configuration in the environment, so that the published images can be used in deployments by setting environment variables to deployment-specific values. Here are some of the images used in the Kuma project:

kuma

The kuma Docker image builds on the kuma_base image, installing a kuma branch and building the assets needed for running as a webservice. The environment can be customized for different deployments. The image can be recreated locally with make build-kuma. The image tagged latest is used by default for development. It can be created locally with make build-kuma VERSION=latest. The official latest image is created from the master branch in Jenkins and published to Docker- Hub. kuma_base

The kuma_base Docker image contains the OS and libraries (C, Python, and Node.js) that support the kuma project. The kuma image extends this by installing the kuma source and building assets needed for production. The image can be recreated locally with make build-base. The image tagged latest is used by default for development. It can be created locally with make build-base VERSION=latest. The official latest image is created from the master branch in Jenkins and published to Docker- Hub kumascript

The kumascript Docker image contains the kumascript rendering engine and support files. The image must be created from the kuma repo since it depends upon that repo’s locale sub-module. From the kuma repo’s root directory, use make build-kumascript for an image tagged with the current commit-hash, make build-kumascript KS_VERSION=latest for one tagged with latest, or make build-kumascript-with-all-tags for an image tagged with both. The image tagged latest is used by default for development, while an image tagged with a commit-hash can be used for deployment. The official images are created from the master branch in Jenkins and published to DockerHub.

2.14 Deploying MDN

MDN is served from Amazon Web Services in the US West (Oregon) datacenters. MDN is deployed as Docker containers managed by Kubernetes. The infrastructure is defined within MDN’s infra repository, including the tools used to deploy and maintain MDN. Deploying new code to AWS takes several steps, and about 60 minutes for the full process. The deployer is responsible for emergency issues for the next 24 hours. There are 1-3 deployments per week, usually between 7 AM and 1 PM Pacific, Monday through Thursday.

Note: This page describes deployment performed by MDN staff. It requires additional setup and permissions not described here. Deployments will not work for non-staff developers, and should not be attempted.

46 Chapter 2. Getting started Kuma Documentation, Release latest

2.14.1 Pre-Deployment

Before deploying, a staff member should: • (Once a week) Check that the latest mdn-browser-compat-data node package is included in the KumaScript image. The latest package is installed when the image is created, during the RUN npm update step, and can be verified by reading the build logs or running the image. • (Once a week) Update the locale and kumascript submodules. The locale submodule must be updated to apply new translations to production. The kumascript submodule is not used for deployment, but updating it keeps the development environment in sync. – Commit or stash any changes in your working copy. – Create a pre-push branch from master: git fetch origin git checkout -b pre-push-`date +"%Y-%m-%d"` origin/master

– Update the submodules: git submodule update --remote git commit -m "Updating submodules" kumascript locale

– Push new localizable strings. This step is only necessary when there are new strings. The TravisCI job TOXENV=locales checks for new strings, and you can examine the output of the recent master build to see if there are differences. At the same time, it’s not too hard to run locally and then revert if there are no useful changes. 1. On the host system, checkout the master locale branch:

cd locale git checkout master git pull

2. Update kuma/settings/common.py, and bump the version in PUENTE[’VERSION’]. 3. Inside the development environment, extract and rebuild the translations:

make localerefresh

4. On the host system (still in the locale subfolder), review the changes to source English strings:

git diff templates/LC_MESSAGES

5. If there are useful changes (ignore comment lines and look for msgid as the changed line), commit the files in the locale submodule:

git add --all . git commit

For the commit message, use the PUENTE[’VERSION’] in the commit subject, and summarize the string changes in the commit body, like:

Update strings 2018.08

* Add "View All" text for document history pagination

Attempt to push to the mdn-l10n repository:

2.14. Deploying MDN 47 Kuma Documentation, Release latest

git push

If this fails, do not force with –force, or attempt to pull and create a merge commit. Someone has added a translation while you were working, and you need to start over to preserve their work:

git fetch git reset --hard @{u}

This resets your locale submodule to the new master. Start over on step 3 (make localerefresh). If the push to mdn-l10n is a success, commit your Kuma changes:

cd .. git commit kuma/settings/common.py locale

You can use the same commit message used for the locale commit. 6. If there are no new strings, throw the changes away, starting in the locale folder:

git checkout -- . cd .. git checkout -- kuma/settings/common.py

– Push the branch and open a pull request: git push -u origin pre-push-`date +"%Y-%m-%d"`

– Check that tests pass. The TOXENV=locales job, which uses Dennis to check strings, is the job that occasionally fails. This is due to a string change committed in Pontoon that will break rendering in produc- tion. If this happens, fix the incorrect translation, push to master on mdn-l10n, and update the submodule in the pull request. Wait for the tests to pass. – Merge the pull request. A “Rebase and merge” is preferred, to avoid a merge commit with little long-term value. Delete the pre-push branch. • Check the latest images from master are built by Jenkins(Kuma master build, KumaScript master build, both private to staff), and uploaded to the DockerHub repositories (Kuma images, KumaScript images).

2.14.2 Deploy to Staging

The staging site is located at https://developer.allizom.org. It runs on the same Kuma code as production, but against a different database, other backing services, and with less resources. It is used for verifying code changes before pushing to production. • Start the staging push, by updating and pushing the stage-push branches: git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd ..

• Prepare for testing on staging:

48 Chapter 2. Getting started Kuma Documentation, Release latest

– Look at the changes to be pushed (What’s Deployed on Kuma, and What’s deployed on KumaScript). To enlist the help of pull request authors and others, you can report bug numbers and PRs in Matrix. – Think about manual tests to confirm the code changes work without errors. • Merge and push to the stage-integration-tests branch: git checkout stage-integration-tests git merge --ff-only origin/master git push

This will kick off functional tests in Jenkins, which will also report to #mdndev. • Manually test changes on https://developer.allizom.org. Look for server errors on homepage and article pages. Try to verify features in the newly pushed code. Check the functional tests. • Announce in Slack (#mdn-dev) that staging looks good, and you are pushing to production.

2.14.3 Deploy to Production

The production site is located at https://developer.mozilla.org. It is monitored by the development team and MozMEAO. • Pick a push song on https://www.youtube.com. Post link to Matrix. • Start the production push: git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd ..

• For the next 30-60 minutes, – Watch https://developer.mozilla.org – Monitor MDN in New Relic for about an hour after the push, for increased errors or performance changes. – Start the standby environment deployment – Close bugs that are now fixed by the deployment – Move relevant Taiga cards to Done – Move relevant Paper cut cards to Done

2.14.4 Deploy to Standby Environment

The standby environment is located in the AWS EU Frankfurt datacenter. It runs the same code and database as production, but runs in read-only maintenance mode and on minimal resources. It will be scaled up and handle MDN traffic if there is a critical failure in the AWS US West datacenter. • Start the standby environment push:

2.14. Deploying MDN 49 Kuma Documentation, Release latest

git fetch origin git checkout standby-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout standby-push git merge --ff-only origin/master git push cd ..

2.15 Rendering Documents

Kuma’s wiki format is a restricted subset of HTML, augmented with KumaScript macros {{likeThis}} that render to HTML for display. Rendering is done in several steps, many of which are stored in the database for read-time performance.

A revision goes through several rendering steps before it appears on the site: 1. A user submits new content, and Kuma stores it as Revision content 2. Kuma bleaches and filters the content to create Bleached content 3. KumaScript renders macros and returns KumaScript content

50 Chapter 2. Getting started Kuma Documentation, Release latest

4. Kuma bleaches and filters the content again to create Rendered content 5. Kuma divides and processes the content into Body HTML, Quick links HTML, ToC HTML, Summary text and HTML There are other rendered outputs: • Kuma normalizes the Revision content into the Diff format to comparing revisions • Kuma filters the Revision content and adds section IDs to create the Triaged content for updating pages • KumaScript renders the Triaged content as Preview content • Kuma filters the Bleached content and adds section IDs to publish the Raw content • Kuma extracts code sections from the Rendered content to flesh out a Live sample • Kuma extracts the text from the Rendered content for the Search content

2.15.1 Revision content

CKEditor provides a visual HTML editor for MDN writers. The raw HTML returned from CKEditor is stored in the Kuma database for further processing. source User-entered content, usually via CKEditor from the edit view (URLs ending with $edit) Developer-submitted content via an HTTP PUT to the API view (URLs ending in $api) displayed on MDN “Revision Source” section of the revision detail view (URLs ending with $revision/), in a

 tag database wiki_revision.content code kuma.wiki.models.Revision.content To illustrate rendering, consider a new document published at /en-US/docs/Sandbox/simple with this Revision content: 
{{ CSSRef }}
I am a simple document with a CSS sidebar.
I am red.
Some Links
 
The HTML Reference
 
{{HTMLElement('div')}}
 
A new document
 
This document has elements that highlight different areas of rendering: • A sidebar macro CSSRef, which will be rendered by KumaScript and extracted for display •A 
 tag, which will gain an id attribute • A list of three links:
2.15. Rendering Documents 51 Kuma Documentation, Release latest
1. An HTML link to an existing document 2. A reference macro HTMLElement which will be rendered by KumaScript 3. An HTML link to a new document, which will get rel="nofollow" and class="new" at- tributes • An onclick attribute, added in Source mode, which will be removed •A   
The Revision content is normalized using pytidylib, a Python interface to the C tidylib library, which turns the content into a well-structured HTML 4.01 document. Content difference reports, or “diffs”, are generated by a line-by-line comparison of the content in Diff format of two revisions. Lines that differ are dropped, so that the reports focus on just the changed content, often without the wrapping HTML tags like . These diffs often contain line numbers from the Diff format, which do not correspond to the line numbers in the Revision content because of differences in formatting and whitespace. Because the Diff format can contain unsafe content, it is not displayed directly on MDN. On Revision comparison views, the Revision dashboard, and in feeds, two Diff formats are processed by difflib.HtmlDiff to generate an HTML 
 showing only the changed lines, and with HTML escaping for the content. For emails, difflib.unified_diff generates a text-based difference report, and it is sent as a plain-text email without escaping.
2.15.10 Triaged content
When a document is re-edited, the Revision content of the current revision is processed before being sent to the editor. This is a lighter version of the full bleaching process used to create Bleached content and Rendered content. source Revision content, with further processing in RevisionForm. displayed on MDN Editing  in the edit view (URLs ending with $edit) Editing <textarea in the translate view (URLs ending with $translate) database not stored</p><p>58 Chapter 2. Getting started Kuma Documentation, Release latest</p><p> code not available For the simple document, this is the Triaged content: <p>{{ CSSRef }}</p></p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p><p style="color:red">I am red.</p></p><p><h2 id="Some_Links">Some Links</h2></p><p><ul> <li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li>{{HTMLElement('div')}}</li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p><script> alert('How about this?'); </script></p><p>The headers get IDs, based on the content, if they did not have them before. For example, id="Some_Links" is added to the <h2>. A simple filter is applied that strips any attributes that start with on, such as the scripting attempt onclick. Further bleaching, for example to remove the <script>, is not applied. CKEditor will perform additional parsing and formatting at load time. It will sometimes notice the empty <div> and replace it with <div class="button"> </div>, especially if it is the last element on the page. It may also remove the <script> element entirely. If a writer makes a change, these backend and CKEditor changes will be reflected in the new Revision content. This can confuse writers (“I didn’t add those IDs!”).</p><p>2.15.11 Preview content</p><p>When editing, a user can request a preview of the document. This sends the in-progress document to editing, with a smaller list of environment variables. source Triaged content, with CKEditor parsing, passed through KumaScript output HTML content at /<locale>/docs/preview-wiki-content database not stored code not available The Preview content for the simple document is: </p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p><p style="color: red;">I am red.</p></p><p><h2>Some Links</h2></p><p><ul></p><p>2.15. Rendering Documents 59 Kuma Documentation, Release latest</p><p><li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li><a href="/en-US/docs/Web/HTML/Element/div" title="The HTML Content Division element (<div>) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."><code><div></code></a></li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p><script> alert('How about this?'); </script></p><p>Fewer environment variables are passed to the KumaScript server for preview than when generating the KumaScript content: url The base URL of the website, like https://developer.mozilla.org/ locale The locale of the request, like "en-US" Some macros use the absence of an environment variable to detect preview mode, and change their output. For example, {{CSSRef}} notices that env.slug is not defined, and outputs an empty string, leaving in the preview output. Other macros don’t have specific code to detect preview mode, and have KumaScript rendering errors in preview. Some macros, like {{HTMLElement}}, work as expected in preview.</p><p>2.15.12 Raw content</p><p>A ?raw parameter can be added to the end of a document to request the source for a revision. This is processed in a similar way to the Triaged content, but from the Bleached content. source Bleached content, with filters output The page with a ?raw query parameter database not stored code not available For the simple document, this is the raw content: <p>{{ CSSRef }}</p></p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p><p style="color: red;">I am red.</p></p><p><h2 id="Some_Links">Some Links</h2></p><p><ul> <li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li>{{HTMLElement('div')}}</li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p><script> alert('How about this?'); </script></p><p>60 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>The Bleached content is parsed for filtering . The headers get IDs, based on the content, if they did not have them before. For example, id="Some_Links" is added to the <h2>. A simple filter is applied that strips any attributes that start with on, such as the scripting attempt onclick. However, this step should do nothing, since these attribute are dropped when creating the Bleached content.</p><p>2.15.13 Live sample</p><p>Live samples are stored in document content. The content is then processed to extract the CSS, JS, and HTML, and reformat them as a stand-alone HTML document suitable for displaying in an <iframe>. source A section extracted from Rendered content, with further processing output Live sample documents on a separate domain, such as https://mdn.mozillademos.org database Not stored in the database, but cached code kuma.wiki.Document.extract.code_sample(section_id) The simple document does not include one of these samples. The Live samples page on MDN describes how the system works for content authors, and includes a live sample demo. Most live samples are loaded in an <iframe>, inserted by the macro EmbedLiveSample. If the sample doesn’t work as an <iframe>, LiveSampleLink can be used instead. The <iframe src= URL is Kuma, running on a different domain, such as https://mdn.mozillademos.org, and configured to serve live samples (the code sample view) and attachments. A separate domain for user-created content, often served in an <iframe>, mitigates many security issues. The live sample is cached on first access, and generated when requested. The extractor looks for <pre> sections with class="brush: html", "brush: css", and "brush: js", to find the sample content, and then selectively un-escapes some HTML and CSS. These sections are used to populate a basic HTML file. There are other sample types that are not derived from wiki content. These are out-of-scope for this document, but the most significant are listed here for the curious: • Legacy samples, like cssref/background-attachment.html, are no longer maintained and are planned for removal (see bug 1076893 and related bugs). • GitHub Live Samples, like the CSS circle demo, are maintained in an MDN repo like mdn/css-examples, served by GitHub pages, and inserted with EmbedGHLiveSample. • Interactive examples are sourced in the mdn/interactive-examples repository, deployed as a static website, inserted with the EmbedInteractiveExamples macro near the top of the page, and are displayed in an <iframe>.</p><p>2.15.14 Search content</p><p>Wiki documents are converted to a JSON format and indexed by ElasticSearch for internal search. This allows search- ing for words in the wiki content. source text extracted from Rendered content output text sections from in-content results from internal search database stored in ElasticSearch code kuma.wiki.search.WikiDocumentType.from_django(Document) The Django utility strip_tags is used to quickly remove HTML tags. This utility is not guarenteed to generate an HTML safe string, as highlighted in a security advisory. Kuma does not redisplay this string. ElasticSearch applies the HTML Strip Char Filter to this and other content, which also strips tags and replaces HTML entities like & with the character equivalents like &.</p><p>2.15. Rendering Documents 61 Kuma Documentation, Release latest</p><p>When a search result is picked because of a content match, ElasticSearch returns the matching section, highlighting the matching terms in bold. This HTML is redisplayed on the search results page. Documents are indexed when created and when updated, as an asyncronous process. Documented are removed from the index when deleted. Administrators can also re-create the entire index, for ElasticSearch upgrades or to freshen the data. There is additional page metadata sent to ElasticSearch to power internal search. This includes page titles, tags, and locales. It also includes KumaScript macros, CSS class names, and HTML attributes, to allow advanced search queries and to power the macro dashboard.</p><p>2.15.15 Future Changes</p><p>Rendering evolved over years, and this document describes how it works, rather than how it was designed. There are some potential changes that would simplify rendering: • Sidebar macros are heavy users of API data and require post-processing of the Rendered content. Sidebar generation could be moved into Kuma instead of being specified by a macro. • The Diff format could be replaced by the Bleached content format, which would be stored for each revision rather than just for the most recent document. • Content from editing could be normalized and filtered before storing as the Revision content. This may unify the Triaged content, Diff format, and Bleached content. • The views that accept new revisions could add IDs to the content before storing the Revision content, rather than wait for the Triaged content or Body HTML. • Developers could refactor the code to consistently access and generate content, rather than repeat filter logic in different forms and views.</p><p>2.15.16 History</p><p>MDN has used different rendering processes in the past. Prior to 2004, Netscape’s DevEdge was a statically-generated website, with content stored in a revision control system (CVS or similar). This was shut down for a while, until Mozilla was able to negotiate a license for the content. From 2005 to 2008, MediaWiki was used as the engine of Mozilla Developer Center. The DevEdge content was converted to MediaWiki Markup. From 2008 to 2011, MindTouch DekiWiki was used as the engine. MindTouch migrated the MediaWiki content to the DekiWiki format, a restricted subset of HTML, augmented with macros (“DekiScript”). During this period, the site was rebranded as Mozilla Developer Network. In 2011, Kuma was forked from Kitsune, the Django-based platform for support.mozilla.org. The wiki format was as close as possible to the DekiWiki format. A new service KumaScript was added to implement DekiScript-style macros. The macros, also known as templates, were stored as content in the database. The service had a GET API to render pages, and a POST API to render previews. In 2013, content zones were added, which allowed a “zone” of pages to have a different style from the rest of the site. For example, the <a href="/tags/Firefox/" rel="tag">Firefox</a> Zone of all the documents under /Mozilla/Firefox had a logo and a shared sub-navigation sidebar. Sub-navigation was similar to quick links, identified by <section id="Subnav">, but stored on the “zone root” (/Mozilla/Firefox) rather than generated by a macro. Zones were part of an effort to consolidate developer documentation on MDN. In 2016, the macros were exported from the Kuma database into the macros folder in the KumaScript repository. The historical changes were exported to mdn/archived_kumascript. This made rendering faster, and allowed code reviews and automated tests of macros, at the cost of requiring review and a production push to deploy macro changes.</p><p>62 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>In 2018, the content zones feature was removed. This was part of an effort to focus MDN Web Docs on common web platform technologies, and away from Mozilla-specific documentation. The sub-navigation feature was dropped. In 2019, the KumaScript engine and macros were modernized to use current features of JavaScript, such as async / await, rather than libraries common in 2011. The API was also unified, so that both previews and standard renders required a POST.</p><p>2.16 Building, collecting, and serving assets</p><p>MDN requires static assets (images, JavaScript, CSS, and other files) to augment the HTML generated by Django. Kuma uses a combination of technologies and techniques to build and deliver static assets. Many of the build and development processes are documented on the Installation and Development documents. This document goes into the details. The three phases of an asset’s life are Building, Collecting, and Serving. • Building - Source files are processed to build intermediate and final assets – Locale files contain translatable strings extracted from source files, are translated by humans, and are compiled to a binary representation for runtime use. – Localization JavaScript files contain translated strings in an executable format to allow in-browser translation of UI strings. – CKEditor is packaged with custom plugins for MDN’s use cases. – Many JavaScript libraries that are used on MDN are included in Kuma’s repository for version con- trol. – JavaScript bundles, assembled by django-pipeline, combine several source files into a single minified JavaScript file. – CSS bundles, assembled by django-pipeline or process:sass, combine several source Sass files into a single minified CSS file. • Collecting - Built assets are collected to the /static folder. – Django’s staticfiles provides a framework of Finder and Storage classes for collecting files. – django-pipeline augments staticfiles, to allow compiling, bundling, and minifiying JS and CSS assets in the collection phase. • Serving - Collected assets are served to visitors in development and production – In Django and Jinja templates, the static tag returns the URL of assets collected by staticfiles. – The statici18n tag returns the URL of localization JavaScript collected by staticfiles. – The pipeline tags javascript and stylesheet return the HTML markup for assets processed and collected by django-pipeline. – WhiteNoise serves static assets as part of the Django process.</p><p>2.16.1 Extracting and building locale files</p><p>Kuma uses Pontoon to translate strings in the user interface, in error messages, and in emails. These are stored in the mdn-l10n repository, and included as a git submodule at locale/. See the localization document for more details about locales.</p><p>2.16. Building, collecting, and serving assets 63 Kuma Documentation, Release latest</p><p>Puente extracts strings to the Portable Object Template (.pot) files, as specified in the PUENTE configuration. The file locale/templates/LC_MESSAGES/django.pot contains strings from template files and Python code. The file javascript.pot contains strings from JavaScript files. Puente looks for the string parameters of gettext functions, such as gettext(), the common alias _(), and ngettext(). It also parses longer strings in the template tag trans. Next the changes are merged into the existing Portable Object (.po) files, such as locale/fr/LC_MESSAGES/django.po, to add new strings and comment out removed strings. Extracting and merging is done with make localeextract, usually during deployment, when UI strings change. This uses the extract management command provided by Puente, which uses Babel to extract strings and update the catalog. A maintainer pushes the updated catalogs as a new commit to the mdn-l10n repository. Pontoon detects that the repository has changed, and notifies localization teams that there are new strings. In about 48 hours, the most active teams will translate strings into the top 10 MDN languages. These are applied by updating the locale submodule during the deployment process. At run time, Machine Object (.mo) files, such as locale/fr/LC_MESSAGES/django.mo, are used by gettext functions, like gettext() and _(), to display the localized strings. These are built with make localecompile when creating the production images or when a developer wants to see updated translations.</p><p>2.16.2 Building localization JavaScript</p><p>Django includes a JavaScriptCatalog view that provides JavaScript implementations of gettext functions, as well as translations for each locale. It is ineffecient to use this view directly, since it is generated on access. For efficiency, django-statici18n generates files for each locale from the JavaScriptCatalog output, so they can be served as static assets. The translation catalog files are created with make compilejsi18n from the locale Machine Object .mo files. This calls the compilejsi18n management command provided by django-statici18n. Kuma sets STATICI18N_ROOT to build/locale, and the output files have names like build/locale/jsi18n/de/javascript.js.</p><p>2.16.3 Building CKEditor</p><p>CKEditor is a complex JavaScript application that provides a WYSIWYG editor for MDN wiki pages. It is packaged with plugins, some from third parties, and some custom to MDN. The CKEditor build process is documented on the CKEditor document. The built files are checked into the Kuma repository.</p><p>2.16.4 Including JS libraries</p><p>Third-party JavaScript libraries are included in the Kuma repository, to avoid ambiguity about what versions of li- braries are used. Some libraries were added manually, and others with Bower. See Front-end asset dependencies for more details about these libraries. Some of these libraries are served directly to visitors, while others are included in pipleline JavaScript bundles.</p><p>2.16.5 Building pipeline JavaScript bundles</p><p>Pipeline JavaScript bundles combine several JavaScript files into a single file, with optional minimization. For exam- ple, the file static/build/js/main.js is the combination of 10 JavaScript files: • kuma/static/js/libs/jquery/jquery.js (JQuery 2.2.0)</p><p>64 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>• kuma/static/js/libs/icons.js • kuma/static/js/components.js • kuma/static/js/analytics.js • kuma/static/js/main.js • kuma/static/js/components/nav-main-search.js • kuma/static/js/auth.js • kuma/static/js/highlight.js • kuma/static/js/wiki-compat-trigger.js • kuma/static/js/lang-switcher.js The JS bundles are specified in PIPELINE_JS in the Django settings. The bundles are served differently in “de- velopment” and “production” modes. This is roughly controlled by the Django setting DEBUG, which sets further parameters like PIPELINE[PIPEINE_ENABLED], and the environment setting DJANGO_SETTINGS_MODULE, which switches the Django settings file. See django-pipeline as well as the pipeline tags section for details. In development, the source files (10 for main.js) are served, so there are 10 <script> elements in the HTML when {{javascript(’main’)}} is used in a template. In production, the output bundle is used, so a single <script> tag appears in the HTML. The single bundle is also processed with UglifyJS, which removes whitespace, replaces variable names with shorter names, and performs other transformations to make the file smaller.</p><p>2.16.6 Building pipeline CSS bundles</p><p>Pipeline CSS bundles are conceptually similar to Pipeline JS Bundles. Some contain multiple source files, such as static/build/styles/dashboards.css, which combines: • kuma/static/styles/dashboards.scss • kuma/static/styles/diff.scss Source styles are written in Sass, and compiled to CSS with node-sass. These must be compiled to CSS in both development and production modes. Backend developers tend to use make build-static to build and collect these files, and front-end developers tend to use nom run process:sass to directly compile them. The CSS bundles are specified in PIPELINE_CSS in the Django settings. The bundles are served differently in “development” and “production” modes. This is roughly controlled by the Django setting DEBUG, which sets further parameters like PIPELINE[PIPEINE_ENABLED], and the environment setting DJANGO_SETTINGS_MODULE, which switches the Django settings file. See django-pipeline as well as the pipeline tags section for details. In development, the source files (2 for dashboards.css) are used, so there are 2 <link> elements in the HTML when when {{stylesheet(’dashboards’)}} is used in a template. In production, the output bundle is used, so a single <link> tag appears in the HTML. When bundled, CSS is also processed by clean-css, which transforms the CSS to make the output files smaller.</p><p>2.16.7 Collecting asset files with staticfiles</p><p>Django provides the django.contrib.staticfiles app, widely used in Django projects to standardize where assets are stored, to collect them for development and production, and to use different asset URLs in different environments. In development mode, the staticfiles app helps identify assets spread across the project, and often allows a rapid development cycle (for example, change a file, refresh the browser, and see the effects of the changed file). For production, the staticfiles app provides the management command collectstatic, which gathers files to the /static folder for efficent file serving.</p><p>2.16. Building, collecting, and serving assets 65 Kuma Documentation, Release latest</p><p>The Django documents for staticfiles are mostly focused on usage. Additional details are needed to understand how django-pipeline customizes staticfiles.</p><p>Configuration</p><p>The staticfiles app is configured by Django settings: STATIC_ROOT The folder on the file system where assets are collected. For MDN, this is the static folder in the kuma directory. STATIC_URL The base URL for static assets. In development, this is http://localhost:8000/static/, and in production it is https://developer.mozilla.org/static/. STATICFILES_FINDERS The dotted path to classes implementing staticfiles Finder. These determine what files will be collected and served. Kuma uses four finders: • django.contrib.staticfiles.finders.FileSystemFinder: Finds files in folders specified by STATICFILES_DIRS • django.contrib.staticfiles.finders.AppDirectoriesFinder: Finds files in the static subfolder of any in- stalled apps • pipeline.finders.CachedFileFinder: Strips hashes from filenames to identify the “pre-cached” names for files. • pipeline.finders.PipelineFinder: When combined assets are not enabled (PIPELINE[’PIPELINE_ENABLED’] == False), returns the source files instead of the combined bundle file. STATICFILES_DIRS A list of folders in the kuma directory that the FileSystemFinder will scan for static assets. For MDN, this includes: • assets/static • assets/ckeditor4/build (to /static/js/libs/ckeditor4/build) • kuma/static • kuma/javascript/dist • build/locale • jinja2/includes/icons For example, the localization JavaScript build/locale/jsi18n/fr/javascript.js will be collected to static/jsi18n/fr/javascript.js. STATICFILES_STORAGE The dotted path to a class implementing staticfiles Storage. Storage determines where files are stored, what URLs they have, and provides hooks for modifying files when copying them. Kuma uses three different storages, depending on the context: • Development server (DEBUG=True): pipeline.storage.NonPackagingPipelineStorage, which avoids com- bining files when collecting them. • Production server (DEBUG=False): kuma.core.pipeline.storage.ManifestPipelineStorage, which com- bines packaged files, hashes the names, and creates a manifest. • Testing (pytest, TravisCI, etc.) and make commands: pipeline.storage.PipelineStorage, which com- bines packaged files but does not hash the names.</p><p>66 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>Finder classes</p><p>The staticfiles app uses Finders to locate asset files. Django considers this a private API, so it may change in the future. There are two methods the BaseFinder class expects to be implemented: • find(path): Given a short path like css/wiki.css, return the absolute path to the file. This is used by the findstatic management command, and to find files when serving assets in development mode. • list(ignore_patterns): Return a list of the files this Finder can find, along with a storage instance for each. The collectstatic management command uses this to gather files. The staticfiles app provides two finders used by Kuma: • The FileSystemFinder collects files under the folders specified in the STATICFILES_DIRS setting. • The AppDirectoriesFinder collects files in the (optional) static subfolder of any installed app listed in INSTALLED_APPS. This is how Django applications, including ones bundled with Django, distribute JavaScript, CSS, images, and other assets. It isn’t used for Kuma’s apps. Instead, we’ve standardized on kuma/static and other named paths. The Finders are used by WhiteNoise to determine which file to serve in development mode. The management com- mand findstatic can be used to determine which file is served, such as: $ ./manage.py findstatic -v2 js/main.js</p><p>Found 'js/main.js' here: /app/kuma/static/js/main.js /app/static/js/main.js Looking in the following locations: /app/kuma/static /app/build/locale /app/jinja2/includes/icons /usr/local/lib/python2.7/site-packages/flat/static /usr/local/lib/python2.7/site-packages/django/contrib/admin/static /usr/local/lib/python2.7/site-packages/constance/static /usr/local/lib/python2.7/site-packages/djcelery/static /usr/local/lib/python2.7/site-packages/django_extensions/static /usr/local/lib/python2.7/site-packages/rest_framework/static /usr/local/lib/python2.7/site-packages/debug_toolbar/static /app/static</p><p>When multiple files are found, the first is used. In the above example, /app/kuma/static/js/main.js will be served in development for /static/js/main.js.</p><p>Storage classes</p><p>The staticfiles app uses a Storage class, which extends Django’s Storage class for asset workflows. Django documents how to write a custom storage system, and there are many 3rd-party storage packages for using various cloud providers for file hosting. The configured STATICFILES_STORAGE class is used when collecting files with ./manage.py collectstatic. Django’s standard Storage classes provide methods like delete(), exists(), and size() for implementing file methods, and methods like listdir() for getting lists of files. There is a wide variety of storage backends with different capabilities, and Django allows most methods to raise NotImplementedErrror if an operation is not supported or is too expensive. A staticfiles Storage class extends the standard Storage classes and requires a few more methods, although the exact methods are undocumented. Some are path(name), to turn a relative path to a full path, and url(path),</p><p>2.16. Building, collecting, and serving assets 67 Kuma Documentation, Release latest</p><p> to get the external URL of the file. An optional method, post_process(), can be defined to further process the files, and returns a map of the old paths to the new paths. The default storage, StaticFilesStorage, is based on the standard FileSystemStorage, and copies static files to STATIC_ROOT (the static folder). For the url() method, it prepends the STATIC_URL to the path. ManifestStaticFilesStorage implements the post_process() method to add the MD5 hash of the file’s contents to the filename. This allows these files to be served with very long cache times, since changes will also change the filename. It also requires manipulating the contents so that references to assets within other files, such as a CSS @import statement, are updated to the hashed names. This often requires source files use relative paths like ../img/logo.svg, so that the tool can find the destination file. Because of the intense file processing, ManifestStaticFilesStorage doesn’t support the live updates of de- velopment mode. It requires DEBUG=False, and that ./manage.py collectstatic is run before running the server, or before a server restart. A map of original to hashed names is stored in staticfiles.json, and is read at server startup to determine the hashed names. CachedStaticFilesStorage is similar to ManifestStaticFilesStorage, but stores the filename mapping in the cache. It is slower than staticfiles.json, and is used when write access to the filesystem is forbidden.</p><p>2.16.8 django-pipeline</p><p>The django-pipeline library is used for packing assets. It provides CSS and JavaScript concatenation and com- pression, built-in JavaScript template support, and optional data-URI image and font embedding. It does this by extending and overriding the django-staticfiles app, so that assets are processed with the standard ./manage.py collectstatic command. Kuma uses django-pipeline to: • Compile Sass .sccs files plain CSS with node-sass • Combine multiple JS and CSS files into a single file (“bundle”) in production • Compress CSS files with cleancss • Compress JS files with UglifyJS</p><p>Configuration</p><p>The django-pipeline app is configured with the dictionary PIPELINE. There are many configuration items, some of which are: • PIPELINE_ENABLED: True to concatenate and compress assets (testing and production), and False to skip concatenation and compression. • PIPELINE_COLLECTOR_ENABLED: True to collect assets (testing and production), and False to skip collection and leave them in the source locations. • COMPILERS: A list of CSS compilers. pipeline‘s SASSCompiler in testing and production, and kuma.core.pipeline.sass.DebugSassCompiler (which does nothing, but instead defers to node-sass) in development. The Makefile specifies the testing configuration, so commands like make collectstatic run with PIPELINE_ENABLED and PIPELINE_COLLECTOR_ENABLED. However, they are disabled when running the development server. django-pipeline specifies outputs as a “package”, which specifies one or more inputs, one output, and some optional settings and overrides. PIPELINE[’JAVASCRIPT’] specifies the JavaScript packages, and PIPELINE[’STYLESHEETS’] specifies the Sass/CSS packages.</p><p>68 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>Finders</p><p>Kuma uses two Finders from django-pipeline. CachedFileFinder strips hashes from filenames to iden- tify the “pre-cached” names for files, by removing the middle element of filenames with three dots. This may have been useful in django-pipeline 1.3 or earlier, but it appears to do nothing now, or could potentially do the wrong thing such as resolving bootstrap.min.js as bootstrap.js. PipelineFinder does nothing if PIPELINE[’PIPELINE_ENABLED’] if True (testing and production), and uses the Storage to find files if it is disabled. For Kuma, this means it may find files in the STATIC_ROOT directory. However, since the FileSystemFinder finds most files in kuma/static first, it is doubtful if this Finder ever applies.</p><p>Storage</p><p>Most of the functionality of django-pipeline is implemented as a Storage class, and Kuma uses three different implementations depending on the environment. The simplest storage, used during testing and in the Makefile, is pipeline.storage.PipelineStorage, which extends the staticfiles Storage class StaticFilesStorage, with a post_process step that packages JS and CSS into one-file bundles, according to the PIPELINE configuration. Development uses pipeline.storage.NonPackagingPipelineStorage. This works the same way as PipelineStorage, but avoids creating packages, where several files are combined into one. JavaScript files are served from the source folders, but CSS files need to be compiled from Sass, and are served from the /static folder after collection. When develop- ing style files, a developer needs to run ./manage.py collectstatic to see changes. In production, kuma.core.pipeline.storage.ManifestPipelineStorage is used. This combines the package processing of PipelineStorage with the hashed assets and staticfiles.json of ManifestStaticFilesStorage. These are generated when the production Docker containers are created.</p><p>2.16.9 Template tag static</p><p>Django provides a template tag static that outputs the URL of the static asset for HTML. Without staticfiles installed, it just adds STATIC_URL to the start of the path. With staticfiles, it calls the url(path) method of the Storage class. In production, with ManifestStaticFilesStorage, it uses staticfiles.json to return a URLs with hashes in the name. For example, here is the HTML that includes the Tumbeast in the 404 page: <div id="beastainer">    </div></p><p>2.16.10 Template tag statici18n</p><p>The tag statici18n is provided by django-statici18n. It works like the static tag, outputing the URL of the localization JavaScript. This is included in <body> of all page via the base template, near the bottom: <script src="{{ statici18n(request.LANGUAGE_CODE) }}"></script></p><p>2.16.11 Template tags javascript and stylesheet</p><p> django-pipeline provides two template tags, {% javascript(’bundle’) %} and {% stylesheet(’bundle’) %}, that can inject the <script> and <link> elements into a template.</p><p>2.16. Building, collecting, and serving assets 69 Kuma Documentation, Release latest</p><p>Bundling is controlled by the setting PIPELINE[’PIPELINE_ENABLED’] (False for development, True for production). When bundled, the assets are assumed to be processed and collected, so a single element representing the final asset URL is inserted. When bundling is off, the assets are assumed to still be in the source form, and multiple HTML elements are inserted into the document. These tags look more like Jinja2 calls then HTML, like these tags from the revision dashboard: {% block js %} {% javascript 'jquery-ui' %} {% javascript 'dashboard' %} {% endblock %}</p><p> django-pipeline supports other output formats. For example, the editor-content bundle is processed with the javascript-array template, which converts the URLs to a format that can be injected into a JavaScript array, such as the configuration script: win.mdn.assets = { css: { 'editor-content': [ {%- stylesheet 'editor-content' %} {%- stylesheet 'editor-locale-%s' % LANG %} ], 'wiki-compat-tables': [{% stylesheet 'wiki-compat-tables' %}] }, js: { 'syntax-prism': [{% javascript 'syntax-prism' %}], 'wiki-compat-tables': [{% javascript 'wiki-compat-tables' %}] } };</p><p>2.16.12 Serving assets with WhiteNoise</p><p>WhiteNoise is a static file serving application, and is an alternative to serving static assets with nginx, Apache, or from Amazon S3. On Kuma, it is used to serve static assets in development as well as production. It made it easy to serve HTML and related assets on the same HTTP/2 connection. In development (DEBUG = True) and testing, WhiteNoise is in “autorefresh” mode, and uses the staticfiles-finder. Each web request to /static scans for the file to use, which can be slow, but will catch any changes made to the files. In production (DEBUG = False), the files in STATIC_ROOT (/static) are indexed when the web server starts up. It also determine headers, such as caching headers and the CORS header, that will be sent with the file. This makes it very fast to serve static files, but changes after the web server starts will not be noticed. WhiteNoise provides its own Storage classes, that can compress and cache static asset files. These are currently unused by Kuma, which uses classes based on those provided by django-pipeline.</p><p>2.16.13 Future</p><p>• Ensure files that are not meant for visitors are not collected, to speed up development, collecting, and preparing production images. • Remove the CachedFileFinder and PipelineFinder. • Remove django-pipeline, using webpack on the server as well before running ./manage.py collectstatic. • Add django-webpack-loader or similar to integrate React assets</p><p>70 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>2.16.14 History</p><p>The staticfiles application was probably part of the Kuma project from the beginning in 2011. In the SCL3 datacenter, one of the first steps of a production push was collecting the static files to a directory on a network drive. This was shared between web servers, so that the new assets were immediately avaiable as the new code was deployed. Because of file hashing, it was possible to keep old versions of assets along with new versions. These files were served by Apache. In 2013, staticfiles was used to serve assets in the development <a href="/tags/Vagrant_(software)/" rel="tag">Vagrant</a> environment instead of Apache, so that collectstatic was not needed to see changes. However, CSS files were converted to Stylus that year, which required compilation for development and deployment. In 2015, several changes were made to prepare for the move from SCL3 to AWS. One change was to move assets from the /media folder, which is traditionally used for user uploads, to the /kuma/static folder. Another was adopting django-pipeline to compile assets, and WhiteNoise to serve them in production. In 2017, MDN hosting moved from SCL3 to AWS. Apache was no longer used to serve assets, and WhiteNoise was used in production as well. This dropped the ability to serve old versions of assets, but a CDN with long caching times mitigated issues around deployments. That same year, the CSS sources were converted from Stylus to Sass. In 2019, the development team decided to adopt new tools such as React and Webpack(ADR-004).</p><p>2.16. Building, collecting, and serving assets 71</p>
                </div>
            </article>
        </div>
    </div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.1/jquery.min.js" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script>
    var docId = '5fac0256c6696d77d6a11982e7d4f2e0';
    var endPage = 1;
    var totalPage = 75;
    var pfLoading = false;
    window.addEventListener('scroll', function () {
        if (pfLoading) return;
        var $now = $('.article-imgview .pf').eq(endPage - 1);
        if (document.documentElement.scrollTop + $(window).height() > $now.offset().top) {
            pfLoading = true;
            endPage++;
            if (endPage > totalPage) return;
            var imgEle = new Image();
            var imgsrc = "//data.docslib.org/img/5fac0256c6696d77d6a11982e7d4f2e0-" + endPage + (endPage > 3 ? ".jpg" : ".webp");
            imgEle.src = imgsrc;
            var $imgLoad = $('<div class="pf" id="pf' + endPage + '"><img src="/loading.gif"></div>');
            $('.article-imgview').append($imgLoad);
            imgEle.addEventListener('load', function () {
                $imgLoad.find('img').attr('src', imgsrc);
                pfLoading = false
            });
            if (endPage < 5) {
                adcall('pf' + endPage);
            }
        }
    }, { passive: true });
    if (totalPage > 0)
        adcall('pf1');
</script>
<script>
    var sc_project = 11552861;
    var sc_invisible = 1;
    var sc_security = "b956b151";
</script>
<script src="https://www.statcounter.com/counter/counter.js" async></script>
</html>