Kuma Documentation Release Latest Mozilla

Kuma Documentation Release latest

Mozilla

February 21, 2019

Contents

1 Development 3

2 Getting started 5 2.1 Installation...... 5 2.2 Development...... 9 2.3 Troubleshooting...... 17 2.4 The Kuma test suite...... 18 2.5 Client-side testing...... 20 2.6 Performance testing...... 24 2.7 Feature toggles...... 25 2.8 Celery and async tasks...... 28 2.9 Search...... 31 2.10 Localization...... 32 2.11 CKEditor...... 35 2.12 Getting Data...... 36 2.13 Documentation...... 40 2.14 Docker...... 41 2.15 Deploying MDN...... 42 2.16 Rendering Documents...... 46 2.17 Building, collecting, and serving assets...... 59

i ii Kuma Documentation, Release latest

Kuma is the platform that powers MDN (developer.mozilla.org)

Contents 1 Kuma Documentation, Release latest

2 Contents CHAPTER 1

Development

Code https://github.com/mozilla/kuma Issues P1 Bugs (to be ﬁxed in current or next sprint) P2 Bugs (to be ﬁxed in 180 days) All developer.mozilla.org bugs Pull Request Queues Dev Docs https://kuma.readthedocs.io/en/latest/installation.html CI Server https://travis-ci.org/mozilla/kuma Forum https://discourse.mozilla.org/c/mdn IRC irc://irc.mozilla.org/mdndev http://logs.glob.uno/?c=mozilla%23mdndev (logs) Servers What’s Deployed on MDN? https://developer.allizom.org/ (stage) https://developer.mozilla.org/ (prod)

3 Kuma Documentation, Release latest

4 Chapter 1. Development CHAPTER 2

Getting started

Want to help make MDN great? Our contribution guide lists some good ﬁrst projects and offers direction on submitting code. Contents:

2.1 Installation

Kuma uses Docker for local development and integration testing, and we are transitioning to Docker containers for deployment as well. Current Status of Dockerization: • Kuma developers are using Docker for daily development and maintenance tasks. Staff developers primarily use Docker for Mac. Other staff members and contributors use Docker’s Ubuntu packages. • The development environment can use a lot of resources. On Docker for Mac, the environment runs well with 6 CPUs and 10 GB of memory dedicated to Docker. It can be run successfully on 2 CPUs and 2 GB of memory. • The Docker development environment is evolving rapidly, to address issues found during development and to move toward a containerized design. You may need to regularly reset your environment to get the current changes. • The Docker development environment doesn’t fully support a ‘production-like’ environment. For example, we don’t have a documented conﬁguration for running with an SSL connection. • When the master branch is updated, the kuma_base image is refreshed and published to DockerHub. This image contains system packages and third-party libraries. • Our TravisCI builds include a target that build Docker containers and runs the tests inside. • Our Jenkins server builds and publishes Docker images, and runs integration tests using Docker. • We are documenting tips and tricks on the Troubleshooting page. • Feel free to ask for help on IRC at #mdndev or on discourse.

2.1.1 Docker setup

1. Install the Docker platform, following Docker’s instructions for your operating system, such as Docker for Mac for MacOS, or for your Linux distribution. Linux users will also want to install Docker Compose and follow post-install instructions to conﬁrm that the development user can run Docker commmands. To conﬁrm that Docker is installed correctly, run:

5 Kuma Documentation, Release latest

docker run hello-world

If you ﬁnd any error using docker commands without sudo visit using docker as non-root user. 2. Clone the kuma Git repository, if you haven’t already: git clone --recursive https://github.com/mozilla/kuma.git

If you think you might be submitting pull requests, consider forking the repository ﬁrst, and then cloning your fork of it. 3. Ensure you are in the existing or newly cloned kuma working copy: cd kuma

4. Initialize and customize .env: cp.env-dist.dev.env vim.env # Or your favorite editor

Linux users should set the UID parameter in .env (i.e. change #UID=1000 to UID=1000) to avoid ﬁle permission issues when mixing docker-compose and docker commands. MacOS users do not need to change any of the defaults to get started. Note that there are settings in this ﬁle that can be useful when debugging, however. 5. Pull the Docker images and build the containers: docker-compose pull docker-compose build

(The build command is effectively a no-op at this point because the pull command just downloaded pre-built docker images.) 6. Start the containers in the background: docker-compose up -d

2.1.2 Load the sample database

Download the sample database with either of the following wget or curl (installed by default on macOS) commands: wget -N https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz curl -O https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz

Next, upload that sample database into the Kuma web container with: docker-compose exec web bash -c "zcat mdn_sample_db.sql.gz | ./manage.py dbshell"

(This command can be adjusted to restore from an uncompressed database, or directly from a mysqldump command.) Then run the following command: docker-compose exec web ./manage.py migrate

This will ensure the sample database is in sync with your version of Kuma.

2.1.3 Compile locales

Localized string databases are included in their source form, and need to be compiled to their binary form:

6 Chapter 2. Getting started Kuma Documentation, Release latest

docker-compose exec web make localecompile

Dozens of lines of warnings will be printed: cd locale; ./compile-mo.sh . ./af/LC_MESSAGES/django.po:2: warning: header field 'PO-Revision-Date' still has the initial default value ./af/LC_MESSAGES/django.po:2: warning: header field 'Last-Translator' still has the initial default value ... ./zu/LC_MESSAGES/javascript.po:2: warning: header field 'PO-Revision-Date' still has the initial default value ./zu/LC_MESSAGES/javascript.po:2: warning: header field 'Last-Translator' still has the initial default value

Warnings are OK, and will be ﬁxed as translators update the strings on Pontoon. If there is an error, the output will end with the error, such as: ./az/LC_MESSAGES/django.po:263: 'msgid' and 'msgstr' entries do not both end with '\n' msgfmt: found 1 fatal error

These need to be ﬁxed by a Kuma developer. Notify them in the #mdndev IRC channel or open a bug. You can continue with installation, but non-English locales will not be localized.

2.1.4 Generate static assets

Static assets such as CSS and JS are included in source form, and need to be compiled to their ﬁnal form: docker-compose exec web make build-static

A few thousand lines will be printed, like: ## Generating JavaScript translation catalogs ## processing language en_US processing language af processing language ar ... ## Compiling (Sass), collecting, and building static files ## Copying '/app/kuma/static/img/embed/promos/survey.svg' Copying '/app/kuma/static/styles/components/home/column-callout.scss' Copying '/app/build/locale/jsi18n/fy-NL/javascript.js' ... Post-processed 'build/styles/editor-locale-ar.css' as 'build/styles/editor-locale-ar.css' Post-processed 'build/styles/locale-ln.css' as 'build/styles/locale-ln.css' Post-processed 'build/styles/editor-locale-pt-BR.css' as 'build/styles/editor-locale-pt-BR.css' .... 1870 static files copied to '/app/static', 125 post-processed.

2.1.5 Visit the homepage

Open the homepage at http://localhost:8000 . You’ve installed Kuma!

2.1.6 Prepare for front-end development (optional)

When doing front-end development on your local machine, you’ll probably want to run gulp, to rebuild front-end assets as they edited, rather than running make build-static after each change. First, install Node.js v8, using the install instructions for your OS. Next, from the root directory of your Kuma repository, install gulp and dependencies:

2.1. Installation 7 Kuma Documentation, Release latest

npm install

Now, you can run gulp (probably from its own shell): node_modules/.bin/gulp

Alternatively, you can install gulp globally: sudo npm install -g

And then run gulp more simply: gulp

2.1.7 Create an admin user

Many Kuma settings require access to the Django admin, including conﬁguring social login. It is useful to create an admin account with password access for local development. If you want to create a new admin account, use createsuperuser: docker-compose exec web ./manage.py createsuperuser

This will prompt you for a username, email address (a fake address like [email protected] will work), and a password. If your database has an existing account that you want to use, run the management command. Replace YOUR_USERNAME with your username and YOUR_PASSWORD with your password: docker-compose run --rm web ./manage.py ihavepower YOUR_USERNAME \ --password YOUR_PASSWORD

With a password-enabled admin account, you can log into Django admin at http://localhost:8000/admin/login/

2.1.8 Enable GitHub authentication (optional)

To enable GitHub authentication, you’ll need to register an OAuth application on GitHub, with settings like: • Application name: MDN Development for (). • Homepage URL: http://localhost:8000/. • Application description: My own GitHub app for MDN! • Authorization callback URL: http://localhost:8000/users/github/login/callback/. As an admin user, add a django-allauth social app for GitHub: • Provider: GitHub. • Name: MDN Development. • Client id: . • Secret key: . • Sites: Move locahost:8000 from “Available sites” to “Chosen sites”. Now you can sign in with GitHub. To associate your password-only admin account with GitHub:

8 Chapter 2. Getting started Kuma Documentation, Release latest

1. Login with your password at http://localhost:8000/admin/login/. 2. Go to the Homepage at https://developer.mozilla.org/en-US/. 3. Click your username at the top to view your profile. 4. Click Edit to edit your profile. 5. Under My Profiles, click Use your GitHub account to sign in. To create a new account with GitHub, use the regular “Sign in” widget at the top of any page. With social accounts are enabled, you can disable the admin password in the Django shell: docker-compose exec web ./manage.py shell_plus >>> me = User.objects.get(username='admin_username') >>> me.set_unusable_password() >>> me.save() >>> exit()

2.1.9 Interact with the Docker containers

The current directory is mounted as the /app folder in the web and worker containers. Changes made to your local directory are usually reﬂected in the running containers. To force the issue, the containers for speciﬁed services can be restarted: docker-compose restart web worker

You can connect to a running container to run commands. For example, you can open an interactive shell in the web container: docker-compose exec web /bin/bash make bash # Same command, less typing

To view the logs generated by a container: docker-compose logs web

To continuously view logs from all containers: docker-compose logs -f

To stop the containers: docker-compose stop

For further information, see the Docker documentation, such as the Docker Overview and the documentation for your operating system. You can try Docker’s guided tutorials, and apply what you’ve learned on the Kuma Docker environment.

2.2 Development

2.2.1 Basic Docker usage

Edit ﬁles as usual on your host machine; the current directory is mounted via Docker host mounting at /app within various Kuma containers. Useful docker sub-commands:

2.2. Development 9 Kuma Documentation, Release latest

docker-compose exec web bash # Start an interactive shell docker-compose logs web # View logs from the web container docker-compose logs -f # Continuously view logs from all containers docker-compose restart web # Force a container to reload docker-compose stop # Shutdown the containers docker-compose up -d # Start the containers docker-compose rm # Destroy the containers

There are make shortcuts on the host for frequent commands, such as: make up # docker-compose up -d make bash # docker-compose exec web bash make shell_plus # docker-compose exec web ./manage.py shell_plus

Run all commands in this doc in the web service container after make bash.

2.2.2 Running Kuma

When the Docker container environment is started (make up or similar), all of the services are also started. The development instance is available at http://localhost:8000.

2.2.3 Running the tests

One way to conﬁrm that everything is working, or to pinpoint what is broken, is to run the test suite.

Django tests

Run the Django test suite: make test

For more information, see the test documentation.

Front-end tests

To run the front-end (selenium) tests, see Client-side Testing.

Kumascript tests

If you’re changing Kumascript, be sure to run its tests too. See https://github.com/mdn/kumascript.

2.2.4 Front-end development

Assets are processed in several steps by Django and django-pipeline: • Front-end localization libraries and string catalogs are generated based on gettext calls in Javascript ﬁles. • Sass source ﬁles are compiled to plain CSS with node-sass. • Assets are collected to the static/ folder. • CSS, JS, and image assets are included in templates using the {% stylesheet %}, {% javascript %}, and {% static %} Jinja2 macros.

10 Chapter 2. Getting started Kuma Documentation, Release latest

In production mode (DEBUG=False), there is additional processing. See Generating production assets for more information. To rebuild the front-end localization libraries: make compilejsi18n

To rebuild CSS, JS, and image assets: make collectstatic

To do both: make build-static

Compiling on the host system with gulp

make build-static is how assets are built for production. It is also slow for iterative front-end development. With DEBUG=True (the default for local development), Gulp can be used to rebuild as ﬁles are changed, using a parallel workﬂow. If you haven’t already installed Node.js and gulp on your local machine, see Prepare for front-end development (optional). On your local (host) machine, open a new shell and run from the root of the Kuma repository: ./node_modules/.bin/gulp gulp # If installed with --global

This gulp command will do two things. First, it will watch all files under ./kuma/static, and any changed file that is not a Sass file (.scss or .sass) under ./kuma/static/styles will be copied to ./static as is (no compilation will be done). Second, it will watch all files with a .scss extension under ./kuma/static/styles, and any change will trigger a stylelint of the changed file, as well as a recompile of all top-level .scss files. All of the resulting compiled files will then be copied to ./static, and immediately available to your local server. This is still faster than the full make build-static build. When running in production mode (DEBUG=False), assets are only read when the webserver starts, so assets processed by gulp will not appear. See Generating production assets for more information.

Style guide and linters

There is an evolving style guide at https://mdn.github.io/mdn-fiori/, sourced from https://github.com/mdn/mdn-fiori. Some of the style guidelines are enforced by linters. To run stylelint on all .scss files: npm run stylelint gulp css:lint # Alternate on the host system

To run eslint on .js ﬁles: npm run eslint

2.2. Development 11 Kuma Documentation, Release latest

2.2.5 Database migrations

Apps are migrated using Django’s migration system. To run the migrations: ./manage.py migrate

If your changes include schema modiﬁcations, see the Django documentation for the migration workﬂow.

2.2.6 Coding conventions

See CONTRIBUTING.md for details of the coding style on Kuma. New code is expected to have test coverage. See the Test Suite docs for tips on writing tests.

2.2.7 Managing dependencies

Python dependencies

Kuma tracks its Python dependencies with pip. See the README in the requirements folder for details.

Front-end asset dependencies

Front-end dependencies are managed by Bower and checked into the repository. Follow these steps to add or upgrade a dependency: 1. On the host, update bower.json. 2. Start a root Docker container shell docker-compose run -u root web bash 3.( Docker only) In the root container shell, run: apt-get update apt-get install -y git npm install -g bower-installer bower-installer

4. On the host, prepare the dependency to be committed (git add path/to/dependency). Front-end dependencies that are not already managed by Bower should begin using this approach the next time they’re upgraded.

Front-end toolchain dependencies

The Front-end toolchain dependencies are managed by npm, but not checked in to the repository. Follow these steps to add or upgrade a dependency: 1. On the host, update package.json. 2. In the web container, install the new dependencies with make npmrefresh 3. On the host, commit the new package.json and package-lock.json.

12 Chapter 2. Getting started Kuma Documentation, Release latest

2.2.8 Customizing with environment variables

Environment variables are used to change the way different components work. There are a few ways to change an environment variables: • Exporting in the shell, such as: export DEBUG=True; ./manage.py runserver

• A one-time override, such as: DEBUG=True ./manage.py runserver

• Changing the environment list in docker-compose.yml. • Creating a .env ﬁle in the repository root directory. One variable you may wish to alter for local development is DEBUG_TOOLBAR, which, when set to True, will enable the Django Debug Toolbar: DEBUG_TOOLBAR=True

Note that enabling the Debug Toolbar can severely impact response time, adding around 4 seconds to page load time.

2.2.9 Customizing the Docker environment

Running docker-compose will create and run several containers, and each container’s environment and settings are configured in docker-compose.yml. The settings are “baked” into the containers created by docker-compose up. To override a container’s settings for development, use a local override file. For example, the web service runs in container with the default command “gunicorn -w 4 --bind 0.0.0.0:8000 --timeout=120 kuma.wsgi:application”. (The container has a name that begins with kuma_web_1_ and ends with a string of random hex digits. You can look up the name of your particular container with docker ps | grep kuma_web. You’ll need this container name for some of the commands described below.) A useful alternative for debugging is to run a single-threaded process that loads the Werkzeug debugger on exceptions (see docs for run- server_plus), and that allows for stepping through the code with a debugger. To use this alternative, create an override file docker-compose.dev.yml: version: "2.1" services: web: command: ./manage.py runserver_plus 0.0.0.0:8000 stdin_open: true tty: true

This is similar to “docker run -it ./manage.py runserver_plus”, using all the other conﬁguration items in docker-compose.yml. Apply the custom setting with: docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d

You can then add pdb breakpoints to the code (import pdb; pdb.set_trace) and connect to the debugger with: docker attach

To always include the override compose ﬁle, add it to your .env ﬁle:

2.2. Development 13 Kuma Documentation, Release latest

COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

A similar method can be used to override environment variables in containers, run additional services, or make other changes. See the docker-compose documentation for more ideas on customizing the Docker environment.

2.2.10 Customizing the database

The database connection is deﬁned by the environment variable DATABASE_URL, with this default: DATABASE_URL=mysql://root:kuma@mysql:3306/developer_mozilla_org

The format is deﬁned by the dj-database-url project: DATABASE_URL=mysql://user:password@host:port/database

If you conﬁgure a new database, override DATABASE_URL to connect to it. To add an empty schema to a freshly created database: ./manage.py migrate

To connect to the database speciﬁed in DATABASE_URL, use: ./manage.py dbshell

2.2.11 Generating production assets

Setting DEBUG=False will put you in production mode, which adds aditional asset processing: • Javascript modules are combined into single JS files. • CSS and JS files are minifed and post-processed by cleancss and UglifyJS. • Assets are renamed to include a hash of contents by a variant of Django’s ManifestStaticFilesStorage. In production mode, assets and their hashes are read once when the server starts, for efficiency. Any changes to assets require rebuilding with make build-static and restarting the web process. The gulp workflow is not compatible with production mode. To emulate production, and test compressed and hashed assets locally: 1. Set the environment variable DEBUG=False 2. Start (docker-compose up -d) your Docker services. 3. Run docker-compose run --rm -e DJANGO_SETTINGS_MODULE=kuma.settings.prod web make build-static. 4. Restart the web process using docker-compose restart web.

2.2.12 Using secure cookies

To prevent error messages like “Forbidden (CSRF cookie not set.):”, set the environment variable: CSRF_COOKIE_SECURE= false

This is the default in Docker, which does not support local development with HTTPS.

14 Chapter 2. Getting started Kuma Documentation, Release latest

2.2.13 Maintenance mode

Maintenance mode is a special conﬁguration for running Kuma in read-only mode, where all operations that would write to the database are blocked. As the name suggests, it’s intended for those times when we’d like to continue to serve documents from a read-only copy of the database, while performing maintenance on the master database. For local Docker-based development in maintenance mode: 1. If you haven’t already, create a read-only user for your local MySQL database: docker-compose up -d docker-compose exec web mysql -h mysql -u root -p (when prompted for the password, enter "kuma") mysql> source ./scripts/create_read_only_user.sql mysql> quit

2. Create a .env ﬁle in the repository root directory, and add these settings: MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

Using a read-only database user is not required in maintenance mode. You can run in maintenance mode just ﬁne with only this setting: MAINTENANCE_MODE=True

and going with a database user that has write privileges. The read-only database user simply provides a level of safety as well as notiﬁcation (for example, an exception will be raised if an attempt to write the database slips through). 3. Update your local Docker instance: docker-compose up -d

4. You may need to recompile your static assets and then restart: docker-compose exec web make build-static docker-compose restart web

You should be good to go! There is a set of integration tests for maintenance mode. If you’d like to run them against your local Docker instance, first do the following: 1. Load the latest sample database (see Load the sample database). 2. Ensure that the test document “en-US/docs/User:anonymous:uitest” has been rendered (all of its macros have been executed). You can check this by browsing to http://localhost:8000/en-US/docs/User:anonymous:uitest. If there is no message about un-rendered content, you are good to go. If there is a message about un-rendered content, you will have to put your local Docker instance back into non-maintenance mode, and render the document: • Configure your .env file for non-maintenance mode: MAINTENANCE_MODE=False DATABASE_USER=root

• docker-compose up -d • Using your browser, do a shift-reload on http://localhost:8000/en-US/docs/User:anonymous:uitest and then put your local Docker instance back in maintenance mode:

2.2. Development 15 Kuma Documentation, Release latest

• Conﬁgure your .env ﬁle for maintenance mode: MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

• docker-compose up -d 3. Conﬁgure your environment with DEBUG=False because the maintenance-mode integration tests check for the non-debug version of the not-found page: DEBUG=False MAINTENANCE_MODE=True DATABASE_USER=kuma_ro

This, in turn, will also require you to recompile your static assets: docker-compose up -d docker-compose exec web ./manage.py compilejsi18n docker-compose exec web ./manage.py collectstatic docker-compose restart web

Now you should be ready for a successful test run: py.test --maintenance-mode -m "not search" tests/functional --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver

Note that the “search” tests are excluded. This is because the tests marked “search” are not currently designed to run against the sample database.

2.2.14 Serving over SSL / HTTPS

Kuma can be served over HTTPS locally with a self-signed certificate. Browsers consider self-signed certificates to be unsafe, and you’ll have to confirm that you want an exception for this. 1. If you want GitHub logins: • In the Django Admin for Sites, ensure that site #2’s domain is set to developer.127.0.0.1.nip.io. • In GitHub, generate a new GitHub OAuth app for the test SSL domain, modifying the pro- cees at Enable GitHub authentication (optional). When creating the GitHub OAuth app, replace http://localhost:8000 with https://developer.127.0.0.1.nip.io in both URLs. When creating the SocialApp in Kuma, chose the developer.127.0.0.1.nip.io site. 2. Include the SSL containers by updating .env: COMPOSE_FILE=docker-compose.yml:docker-compose.ssl.yml

3. Run the new containers: docker-compose up -d

4. Load https://developer.127.0.0.1.nip.io/en-US/ in your browser, and add an exception for the self-signed certificate. 5. Load https://demos.developer.127.0.0.1.nip.io/en-US/ in your browser, and add an exception for the self-signed certificate again. Some features of SSL-protected sites may not be available, because the browser does not fully trust the self-signed SSL certificate. The HTTP-only website will still be available at http://localhost:8000/en-US/, but GitHub logins will not work.

16 Chapter 2. Getting started Kuma Documentation, Release latest

2.3 Troubleshooting

Kuma has many components. Even core developers need reminders of how to keep them all working together. This doc outlines some problems and potential solutions running Kuma.

2.3.1 Kuma “Reset”

These commands will reset your environment to a “fresh” version, with the current third-party libraries, while retaining the database: cd /path/to/kuma docker-compose down make clean git submodule sync --recursive && git submodule update --init --recursive docker-compose pull docker-compose build --pull docker-compose up

2.3.2 Reset a corrupt database

The Kuma database can become corrupted if the system runs out of disk space, or is unexpectedly shutdown. MySQL can repair some issues, but sometimes you have to start from scratch. When this happens, pass an extra argument --volumes to down: cd /path/to/kuma docker-compose down --volumes make clean git submodule sync --recursive && git submodule update --init --recursive docker-compose pull docker-compose build --pull docker-compose up -d mysql sleep 20 # Wait for MySQL to initialize. See notes below docker-compose up

The --volumes flag will remove the named MySQL database volume, which will be recreated when you run docker-compose up. The mysql container will take longer to start up as it recreates an empty database, and the kuma container will fail until the mysql container is ready for connections. The 20 second sleep should be sufficient, but if it is not, you may need to cancel and run docker-compose up again. Once mysql is ready for connections, follow the installation instructions, starting at Provisioning a database, to configure your now empty database.

2.3.3 Run alternate services

Docker services run as containers. To change the commands or environments of services, it is easiest to add an override conﬁguration ﬁle, as documented in Customizing the Docker environment.

2.3.4 Linux ﬁle permissions

On Linux, it is common that ﬁles created inside a Docker container are owned by the root user on the host system. This can cause problems when trying to work with them after creation. We are investigating solutions to create ﬁles as

2.3. Troubleshooting 17 Kuma Documentation, Release latest

the developer’s user. In some cases, you can specify the host user ID when running commands: docker-compose run --rm --user $(id -u) web ./manage.py collectstatic

In other cases, the command requires root permissions inside the container, and this trick can’t be used. Another option is to allow the ﬁles to be created as root, and then change them to your user on the host system: find . -user root -exec sudo chown $(id -u):$(id -g) \{\} \;

2.3.5 Getting more help

If you have more problems running Kuma, please: 1. Paste errors to pastebin. 2. Start a thread on Discourse. 3. After you email dev-mdn, you can also ask in IRC.

2.4 The Kuma test suite

Kuma has a fairly comprehensive Python test suite. Changes should not break tests. Only change a test if there is a good reason to change the expected behavior. New code should come with tests. Commands should be run inside the development environment, after make bash.

2.4.1 Setup

Before you run the tests, you have to build assets: make build-static

2.4.2 Running the test suite

If you followed the steps in the installation docs, then all you should need to do to run the test suite is: make test

The default options for running the test are in pytest.ini. This is a good set of defaults. If you ever need to change the defaults, you can do so at the command line by running what the Make task does behind the scenes: py.test kuma

Some helpful command line arguments to py.test (won’t work on make test): --pdb: Drop into pdb on test failure. --create-db: Create a new test database. --showlocals: Shows local variables in tracebacks on errors. --exitfirst: Exits on the ﬁrst failure.

18 Chapter 2. Getting started Kuma Documentation, Release latest

See py.test --help for more arguments.

Running subsets of tests and speciﬁc tests

There are a bunch of ways to specify a subset of tests to run: • only tests marked with the ‘spam’ marker: py.test -m spam

• all the tests but those marked with the ‘spam’ marker: py.test -m "not spam"

• all the tests but the ones in kuma/core: py.test --ignore kuma/core

• all the tests that have “foobar” in their names: py.test -k foobar

• all the tests that don’t have “foobar” in their names: py.test -k "not foobar"

• tests in a certain directory: py.test kuma/wiki/

• speciﬁc test: py.test kuma/wiki/tests/test_views.py::RedirectTests::test_redirects_only_internal

See http://pytest.org/latest/usage.html for more examples.

Showing test coverage

While running the tests you can record which part of the code base is covered by test cases. To show the results at the end of the test run use this command: make coveragetest

To generate an HTML coverage report, use: make coveragetesthtml

The test database

The test suite will create a new database named test_%s where %s is whatever value you have for settings.DATABASES[’default’][’NAME’]. Make sure the user has ALL on the test database as well.

2.4.3 Markers

See:

2.4. The Kuma test suite 19 Kuma Documentation, Release latest

py.test--markers

for the list of available markers. To add a marker, add it to the pytest.ini ﬁle. To use a marker, add a decorator to the class or function. Examples: import pytest

@pytest.mark.spam class SpamTests(TestCase): ...

class OtherSpamTests(TestCase): @pytest.mark.spam def test_something(self): ...

2.4.4 Adding tests

Code should be written so that it can be tested, and then there should be tests for it. When adding code to an app, tests should be added in that app that cover the new functionality. All apps have a tests module where tests should go. They will be discovered automatically by the test runner as long as the look like a test.

2.4.5 Changing tests

Unless the current behavior, and thus the test that veriﬁes that behavior is correct, is demonstrably wrong, don’t change tests. Tests may be refactored as long as it’s clear that the result is the same.

2.4.6 Removing tests

On those rare, wonderful occasions when we get to remove code, we should remove the tests for it, as well. If we liberate some functionality into a new package, the tests for that functionality should move to that package, too.

2.5 Client-side testing

Kuma has a suite of functional tests using Selenium and pytest. This allows us to emulate users interacting with a real browser. All these test suites live in the /tests/ directory. The tests directory comprises of: • /functional contains pytest tests. • /pages contains Python PyPOM page objects. • /utils contains helper functions. • /headless contains tests that don’t require a browser

20 Chapter 2. Getting started Kuma Documentation, Release latest

2.5.1 Setting up test virtual environment

1. In the terminal go to the directory where you have downloaded kuma. 2. Create a virtual environment for the tests to run in (the example uses the name mdntests): $ virtualenv mdntests

3. Activate the virtual environment: $ source mdntests/bin/activate

4. Make sure the version of pip in the virtual environment is high enough to support hashes: $ pip install "pip>=8"

5. Install the requirements the tests need to run: $ pip install -r requirements/test.txt

You may want to add your virtual environment folder to your local .gitignore ﬁle.

2.5.2 Running the tests

Before running the tests you will need to download the driver for the browser you want to test in. Drivers are listed and linked to in the pytest-selenium documentation. The minimum amount of information necessary to run the tests against the staging server is the directory the tests are in, which driver to use, and the subset of tests to run (by specifying a marker or marker expression like -m "not login", for more information see Markers). Currently, all of the tests except the tests marked “login” should run successfully against the staging server. In the virtual environment run: $ pytest -m "not login" tests --driver Chrome --driver-path /path/to/chromedriver

You will be prompted “Do you want the application ‘python’ to accept incoming network connections?” The tests seem to run ﬁne no matter how you answer. This basic command can be modiﬁed to use different browsers, to run the tests against a local environment, or to run tests concurrently.

Only running tests in one ﬁle

Add the name of the ﬁle to the test location: $ pytest -m "not login" tests/functional/test_search.py --driver Chrome --driver-path /path/to/chromedriver

Run the tests against a different url

By default the tests will run against the staging server. If you’d like to run the tests against a different URL (e.g., your local environment) pass the desired URL to the command with --base-url: $ pytest -m "not login" tests --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver

2.5. Client-side testing 21 Kuma Documentation, Release latest

Only running headless tests

Headless tests do not require Selenium or markers: $ pytest tests/headless --base-url http://localhost:8000

Run the tests in parallel

By default the tests will run one after the other but you can run several at the same time by specifying a value for -n: $ pytest -m "not login" tests -n auto --driver Chrome --driver-path /path/to/chromedriver

Run the tests against a server conﬁgured in maintenance mode

By default the tests will run against a server assumed to be configured normally. If you’d like to run them against a server configured in maintenance mode, simply add --maintenance-mode to the pytest command line. For example, if you’ve configured your local environment to run in maintenance mode: $ pytest --maintenance-mode -m "not search" tests --base-url http://localhost:8000 --driver Chrome --driver-path /path/to/chromedriver

Note that the tests marked “search” were excluded assuming you’ve loaded the sample database. If you’ve loaded a full copy of the production database, you can drop the -m "not search". The --maintenance-mode command-line switch does two things. It will skip any tests that don’t make sense in maintenance mode (e.g., making sure the signin link works), and include the tests that only make sense in maintenance mode (e.g., making sure that endpoints related to editing redirect to the maintenance-mode page).

2.5.3 Using Alternate Environments

Run tests on SauceLabs

Running the tests on SauceLabs will allow you to test browsers not on your host machine. 1. Signup for an account. 2. Log in and obtain your Remote Access Key from user settings. 3. Run a test specifying SauceLabs as your driver, and pass your credentials and the browser to test: $ SAUCELABS_USERNAME=thedude SAUCELABS_API_KEY=123456789 pytest -m "not login" tests/functional/ --driver SauceLabs --capability browsername MicrosoftEdge

Alternatively you can save your credentials in a conﬁguration ﬁle so you don’t have to type them each time.

Run tests on MDN’s Continuous Integration (CI) infrastructure

If you have commit rights on the mozilla/kuma GitHub repository you can run the UI tests using the MDN CI Infras- tructure. Just force push to mozilla/kuma@stage-integration-tests to run the tests against https://developer.allizom.org. You can check the status, progress, and logs of the test runs at MDN’s Jenkins-based multi-branch pipeline.

22 Chapter 2. Getting started Kuma Documentation, Release latest

Run tests locally using Selenium Docker images

Running Selenium locally will take over your browser, and may make it difﬁcult to do other things with your development system. One option is to use Selenium’s Docker images, which contain browsers, drivers, and the selenium agent.

Note: This feature is in development, and some tests will fail against the local environment.

To run dockerized Selenium against your local development environment: $ scripts/run_functional_tests.sh

This uses the “standalone” Docker images, and runs the tests in Chrome and Firefox. It runs pytest with the options tests/functional -m "not login" -vv --reruns=1 (run functional tests, skip those requiring a login, be very verbose, and try failing tests again). It places test results in a subdirectory of test_results, which includes logs, screenshots, and other details. To run with Selenium Grid, like the MDN CI Infrastructure, use: $ SELENIUM_HUB=1 scripts/run_functional_tests.sh

You can replace the default pytest options by passing arguments: $ scripts/run_functional_tests.sh -m "not login" tests/functional/test_article_edit.py

You can test staging by setting a new base URL as a pytest argument: $ scripts/run_functional_tests.sh --base-url https://developer.allizom.org -m "not login" tests/functional/test_article_edit.py

You can also use an environment variable and get the default pytest arguments: $ BASE_URL=https://developer.allizom.org scripts/run_functional_tests.sh

See scripts/run_functional_tests.sh for the all the conﬁguration options.

2.5.4 MDN CI Infrastructure

The MDN CI infrastructure is a Jenkins-based, multi-branch pipeline. The pipelines for all branches are defined by the Jenkinsfile and the files under the Jenkinsfiles directory. The basic idea is that every branch may have its own custom pipeline steps and configuration. Jenkins will auto-discover the steps and configuration by checking within the Jenkinsfiles directory for a Groovy (.groovy) and/or YAML (.yml) file with the same name as the branch. For example, the “stage-integration-tests” branch has a Jenkinsfiles/stage-integration-tests.yml file which will be loaded as configuration and used to determine what to do next (load and run the Groovy script specified by its pipeline.script setting - Jenkinsfiles/integration- tests.groovy - and the script, in turn, will use the dictionary of values provided by the job setting defined within the configuration). Note that the YAML files for the integration-test branches provide settings for configuring things like the version of Selenium to use, the number of Selenium nodes to spin-up, the Dockerfile to use to build the container, the URL to test against, and which subset of integration tests to run (via a pytest marker expression, see Markers). The integration-test Groovy files use a number of global Jenkins functions that were developed to make the building, running and pushing of Docker containers seamless (as well as other cool stuff, see mozmar/jenkins-pipeline). They allow us to better handle situations that have been painful in the past, like the stopping of background Docker containers.

2.5. Client-side testing 23 Kuma Documentation, Release latest

The “prod-integration-tests” branch also has its own Jenkinsfiles/prod-integration-tests.yml file. It’s identical to the YAML file for the “stage-integration-tests” branch except that it specifies that the tests should be run against the production server rather than the staging server (via its job.base_url setting). Similarly, the “master” branch has it’s own pipeline, but instead of being configured by a YAML file, the entire pipeline is defined within its Jenkinsfiles/master.groovy file. The pipeline for any other branch which does not provide its own Groovy and/or YAML file will follow that defined by the Jenkinsfiles/default.groovy file. You can check the status, progress, and logs of any pipeline runs via MDN’s Jenkins-based multi-branch pipeline.

2.5.5 Markers

• nondestructive Tests are considered destructive unless otherwise indicated. Tests that create, modify, or delete data are considered destructive and should not be run in production. • smoke These tests should be the critical baseline functional tests. • nodata New instances of kuma have empty databases so only a subset of tests can be run against them. These tests are marked with nodata. • login These tests require the testing accounts to exist on the target site. For security reasons these accounts will not be on production. Exclude these tests with -m "not login"

2.5.6 Guidelines for writing tests

See Bedrock and the Web QA Style Guide.

2.6 Performance testing

In 2015, as part of a “measure and improve performance” initiative, MDN staff configured Locust for performance testing of MDN. The locust test files simulate read traffic to MDN, fetching the top pages in roughly the same ratios as they were fetched on MDN, and can be run against test environments to measure changes in code, caching and settings. These tests are not maintained or run during the current development process. Some of the common URLs are not included in the sample database.

2.6.1 Running tests

Note: DO NOT RUN LOCUST TESTS AGAINST PRODUCTION

1. Create a ﬁle docker-compose.locust.yml in the root folder (same folder as docker-compose.yml) with the contents:

24 Chapter 2. Getting started Kuma Documentation, Release latest

version: "2.1" services: locust: build: ./tests/performance depends_on: - web environment: - LOCUST_MODE=standalone - TARGET_URL=http://web:8000 volumes: - ./tests/performance/:/tests ports: - "8089:8089"

2. Edit .env to update docker-compose: COMPOSE_FILE=docker-compose.yml:docker-compose.locust.yml

3. Restart with make up 4. Load the Locust UI at http://localhost:8089, and select: • number of users to simulate • users spawned per second See Start locust for more.

2.6.2 Adding a test suite

See Writing a locustﬁle for more.

2.7 Feature toggles

MDN uses feature toggles to integrate un-finished feature changes as early as possible, and to control the behavior of finished features. Some site features are controlled using django-waffle. You control these features in the django admin site’s waffle section. Some site features are controlled using constance. You control these features in the django admin site’s constance section.

2.7.1 Wafﬂe features

Switches

Wafﬂe switches are simple booleans - they are either on or off. • application_ACAO - Enable Access-Control-Allow-Origin=0 header. • foundation_callout - Show foundation donate homepage tile • helpful-survey-2 - Enable 2017 Helpfulness Survey • registration_disabled - Enable/disable new user registration.

2.7. Feature toggles 25 Kuma Documentation, Release latest

• store_revision_ips - Save request data, including the IP address, to enable marking revisions as spam. • welcome_email - Send welcome email to new user registrations. • wiki_spam_training - Call Akismet to check submissions, but don’t block due to detected spam or Ak- ismet errors.

Flags

Waffle flags control behavior by specific users, groups, percentages, and other advanced criteria. • contrib_beta - Enable/disable the contributions popup and pages • kumaediting - Enable/disable wiki editing. • page_move - (deprecated) enable/disable page move feature. • section_edit - Show section edit buttons. • sg_task_completion - Enable the Survey Gizmo pop-up. • spam_admin_override - Tell Akismet that edits are never spam. • spam_checks_enabled - Toggle spam checks site wide. • spam_spammer_override - Tell Akismet that edits are always spam. • spam_submissions_enabled - Toggle Akismet spam/spam submission ability. • spam_testing_mode - Tell Akismet that edits are tests, not real content.

2.7.2 Constance features

Constance conﬁgs let us set operational values for certain features in the database - so we can control them without changing code. They are all listed and documented in the admin site’s constance section.

2.7.3 Using Trafﬁc Cop for A/B Testing

Traffic Cop is a lightweight JavaScript library that allows conducting content experiments on MDN without the need for third party tools such as Optimizely. Traffic Cop also respects user privacy, and will not run if Do Not Track is enabled. On MDN, Traffic Cop will also not run for logged in users. Here’s how to implement a Traffic Cop content experiment. The first task is to choose your control page and create your tests pages Here we will use the scenario of testing a new feature on MDN. The new feature involved replacing the static examples on top, with an interactive editor. First we choose our control page. For this example, we will use: https://developer.mozilla.org/docs/Experiment:ExperimentName/Array.prototype.push() Next we need our test page. The convention on MDN is to first create a base page that follows a naming convention such as: https://developer.mozilla.org/docs/Experiment:ExperimentName/ With the base page created, we next create out test variation of our chosen page. To do this, navigate to: https://developer.mozilla.org/docs/Experiment:ExperimentName/Array.prototype.push() You should should now be presented with an editing interface to create the new page.

26 Chapter 2. Getting started Kuma Documentation, Release latest

Switch back to the control page, and click the Edit button. Switch the editor to source mode and copy all of the contents inside the editor. Switch back to our test, ensure the editor is set to source mode, and paste the content you just copied. You can now go ahead and make the changes you wish to user test. Once you are complete, go ahead and publish the page.

The JavaScript

Let us first add the JavaScript code to initialise and configure Traffic Cop. Inside kuma\static\js, create a new file called some like experiment-experiment-name.js and paste the following code into the file:

1 (function() {

2 'use strict';

4 var cop= new Mozilla.TrafficCop({

5 id: 'experiment-experiment-name',

6 variations:{

7 'v=a': 50, // control

8 'v=b': 50 // test

9 }

10 });

12 cop.init();

13 })(window.Mozilla);

This will initialise Trafﬁc Cop, set up an experiment with the id experiment-experiment-name, and lastly deﬁne a 50/50 split between the control, and the test page.

Deﬁne your bundle

Next we need to add an entry into kuma\settings\common.py to identify the test, and the related JS that will be injected. Find the following line in common.py: PIPELINE_JS = {

Just before the closing } add a block such as the following:

1 'experiment-experiment-name':{

2 'source_filenames':(

3 'js/libs/mozilla.dnthelper.js',

4 'js/libs/mozilla.cookiehelper.js',

5 'js/libs/mozilla.trafficcop.js',

6 'js/experiment-experiment-name.js',

7 ),

8 'output_filename':'build/js/experiment-experiment-name.js',

9 },

NOTE: The key here experiment-experiment-name needs to match the id you speciﬁed in your JS ﬁle above.

Identify your A/B test pages

The final step is to identify the pages to the back-end so it will know where to direct traffic based on the URL parameter that will be added by Traffic Cop. Inside kuma\settings\content_experiments.json add the following:

2.7. Feature toggles 27 Kuma Documentation, Release latest

1 [

2 {

3 "id": "experiment-experiment-name",

4 "ga_name": "experiment-name",

5 "param": "v",

6 "pages":{

7 "en-US:Web/JavaScript/Reference/Global_Objects/Array/push":{

8 "a": "Web/JavaScript/Reference/Global_Objects/Array/push",

9 "b": "Experiment:ExperimentName/Array.prototype.push()"

10 },

11 }

12 }

13 ]

There are a couple of important points to note here: 1. We are leaving of the domain, as well as the docs/ part of the url. 2. As with the entry in common.py, the id here matches the id in your JS file, tying it all together. 3. The param value, needs to match the string you specified inside the variations block in your JS 4. The first part of our key under pages above, identifies the locale to which this will apply, en-US in this case. 5. The key for the two pages listed next, needs to match the values you used as the parameter value inside variations in your JS file earlier.

Testing your experiment

With you local instance of Kuma running, navigate to the page you defined as your control. In this example: http://localhost:8000/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/push NOTE: You should not be logged in to MDN, and ensure that Do Not Track is disabled. Your experiment JavaScript code bundle defined in common.py should be injected into the page, and Traffic Cop will add a URL parameter to the page that is either v=a or v=b. Depending on which, you will either see the control(a), or the variation(b). You can also force a specific page to load by appending ?v=a or, ?v=a manually to the control page URL. If all the above works as expected, open up a pull request, and tag someone on MDN for reivew.

2.8 Celery and async tasks

Kuma uses Celery to enable asynchronous task processing for long-running jobs and to speed up the request-response cycle.

2.8.1 When is Celery appropriate

You can use Celery to do any processing that doesn’t need to happen in the current request-response cycle. Ask yourself the question: “Is the user going to need this data on the page I’m about to send them?” If not, using a Celery task may be a good choice. Some examples where Celery is used:

28 Chapter 2. Getting started Kuma Documentation, Release latest

• Sending notification emails on page changes - If this was done in the page change request, then pages with many watchers would be slower to edit, and errors in email sending would result in a “500 Internal Server Error” for the editor. • Populating the contributor bar - Gathering and sorting the contributor list can be a slow process that would delay rendering of stale pages. Instead, viewers quickly see the content, possibly without the contributor bar. Once the async task has populated the cache, future viewers get the content and the contributor bar. • Generating the spam moderator dashboard - Calculating spam statistics is a potentially long process, and if it takes more than 30 seconds the viewer will get a “502 Bad Gateway” error. The async task can take as long as needed, and the spam moderator will see the dashboard when the data is calculated and cached. • Rebuilding sitemaps - Search engines need recent site URL data, and it can’t be assembled quickly in the request for the sitemap. An async request repopulates the cached sitemap data, keeping this data fast and up-to- date. There are some downsides to async tasks that should also be considered: • Message passing, serialization, de-serialization, and data reloading increase the load on the entire system. • Async tasks require different testing strategies. It is easy to write a passing test that doesn’t reflect how the task is called in production. • Runtime errors are more difficult to troubleshoot with async tasks, due to missing context on how they were called. In general, it is better to get an algorithm right in the request loop, and only move it to an asynchronous task when it is identified as a performance issue.

2.8.2 Celery services

A working Celery installation requires several services.

Worker

Celery processes tasks with one or more workers. In Kuma, the workers and web processes share a code base, so that Django models, functions, and settings are available to async tasks, and web code can easily schedule async tasks. In Docker, the worker process runs in the worker service / container.

Broker

Celery requires a message broker for task communication. There are two stable, production-ready alternatives: • Redis is an in-memory data structure store, used as database, cache and message broker. Many projects use it for multiple roles in the same deployment. With Celery 3.1, it is now recommended as a stable broker, with some caveats. It is used in the Docker environment. • RabbitMQ is an AMQP message broker written in Erlang. The Celery team has recommended it for a long time, and the docs describe it as “feature-complete, stable, durable and easy to install”. It is used in the production environment.

Result store

When a task completes, it returns processed data and task states to a results store. Kuma doesn’t use returned data, but it does use returned task state to coordinate multi-step tasks.

2.8. Celery and async tasks 29 Kuma Documentation, Release latest

Kuma uses a database-backed task store provided by django-celery, a deprecated integration project for Django and Celery. The work to replace this is tracked in bug 1268257.

Periodic tasks scheduler

Periodic tasks are scheduled with celery beat, which adds tasks to the task queue when they become due. It should only be run once in a deployment, or tasks may be scheduled multiple times. In Docker, it runs in the worker container by starting the celery process with --beat. In production, there are several task workers, and the celery beat process is run directly on just one worker. The schedules themselves are conﬁgured using the deprecated django-celery database backend. The work to replace this is tracked in bug 1268256.

Monitoring

Celery task workers generate events to communicate task and worker health. The celerycam service, provided by the deprecated django-celery project, captures these events and stores them in the database. Switching to a supported project, like Flower, is tracked in bug 1268281. It is not part of the default Docker services, but can be started inside the worker service container with: ./manage.py celerycam --freq=2.0

In production, celerycam is run directly on one worker. For more options, see the Monitoring and Management Guide in the Celery documentation.

2.8.3 Conﬁguring and running Celery

We set some reasonable defaults for Celery in kuma/settings/common.py. These can be overridden by the environment variables, including: • CELERY_ALWAYS_EAGER Default: False (Docker), True (tests). When True, tasks are executed immediately, instead of being scheduled and executed by a worker, skipping the broker, results store, etc. In theory, tasks should act the same whether executed eagerly or normally. In practice, there are some tasks that fail or have different results in the two modes, mostly due to database transactions. • CELERYD_CONCURRENCY Default: 4. Each worker will execute this number of tasks at the same time. 1 is a good value for debugging, and the number of CPUs in your environment is good for performance. The worker can also be adjusted at the command line. For example, this could run inside the worker service container: ./manage.py celeryd --log-level=DEBUG -c 10

This would start Celery with 10 worker threads and a log level of DEBUG.

30 Chapter 2. Getting started Kuma Documentation, Release latest

2.9 Search

Kuma uses Elasticsearch to power its on-site search.

2.9.1 Installed version

Elasticsearch releases new versions often. They release a new minor version (for example, 6.2.0, 6.3.0) every 30 to 90 days, and support the most recent minor version. They release a major version (5.0.0, 6.0.0) every 12 to 18 months, and support the last minor version of the previous major release (for example, 2.4.x supported until 6.x is released, 5.6.x supported until 7.x is released). Kuma’s goal is to update to the last minor version of the previous major release, so we expect to update every 12 to 18 months. Kuma is running on ElasticSearch 5.6.10 in development and production, which is planned to be supported until March 2019. The Kuma search tests use the conﬁgured Elasticsearch, and can be run inside the web Docker container to conﬁrm the engine works: py.test kuma/search/

2.9.2 Indexing documents

To see search results, you will need to index the documents.

Using the Admin

This process works both in a development environment and in production: • Open the Django admin search index list view. On Docker, this is located at http://localhost:8000/admin/search/index/. • Add a search index by clicking on the “Add index” button in the top right corner. Optionally name it, or leave it blank to generate a valid name based on the time. Spaces are not allowed in the name. Click the “SAVE” button in the lower right corner. • On the search index list, select the checkbox next to the newly created index (the top most) and select “Populate selected search index via Celery” from the Action dropdown menu. Click “Go” to start indexing asynchronously. • In the development environment, you can watch the indexing process with: docker-compose logs -f elasticsearch worker

• Refresh the search index list until the “populated” field changes to a green checkbox image. In production, you will also get an email notifying you when the index is populated. • Select the checkbox next to the populated index, then choose “Promote selected search index to current index” in the actions dropdox. Click “Go” to promote the index. • All three fields will be green checkboxes (promoted, populates, and “is current index?”). The index is live, and will be used for site search. Similarly you can also demote a search index and it will automatically fall back to the previously created index (by created date). That helps to figure out issues in production and should allow for a smooth deployment of search index changes. It’s recommended to keep a few search indexes around just in case. If no index is created in the admin UI the fallback “main_index” index will be used instead.

2.9. Search 31 Kuma Documentation, Release latest

When you delete a search index in the admin interface, it is deleted on Elasticsearch as well.

Using the shell

Inside the web Docker container: ./manage.py reindex

This will populate and activate the fallback index “main_index”. It will be overwritten when the search tests run.

2.9.3 Search Filters

The search interface uses Filters and Filter Groups to determine what results are shown. Currently, the only active Filter Group is “Topics”. The Filters collect documents by tag. For example, the Filter “Add-ons & Extensions” collects all documents tagged with “Add-ons”, “Extensions”, “Plugins”, or “Themes”. Most filters have a single associated tag. For example, the “CSS” filter collects the documents with the “CSS” tag. Filter and Filter Groups are defined in the database, but the names are shown to users. The names of these objects (enabled and disabled) are listed in kuma/search/names.py, and marked for translation. To generate this file, run this command against the production database, and copy the output to names.py: ./manage.py generate_search_names

The sample database contains the active subset of Filters and Filter Groups. This won’t match the production database, and so the development environment should not be used to generate kuma/search/names.py.

2.10 Localization

Kuma is localized with gettext. User-facing strings in the code or templates need to be marked for gettext localization. We use Pontoon to provide an easy interface to localizing these ﬁles. Pontoon allows translators to use the web UI as well as download the PO ﬁles, use whatever tool the translator feels comfortable with and upload it back to Pontoon. The strings are stored in a separate repository, https://github.com/mozilla-l10n/mdn-l10n

Note: We do not accept pull requests for updating translated strings. Please use Pontoon instead.

See the Django documentation on Translations for how to make strings marked for translation in Python and templates. Unless otherwise noted, run all the commands in this document inside the development environment. For Docker on Linux, set UID in .env or enter the environment with docker-compose run --rm --user $(id -u) web bash, to ensure that created ﬁles are owned by your development user.

2.10.1 Build the localizations

Localizations are found in this repository under the locale folder. This folder is a git submodule, linked to the mdn-l10n repository. The gettext portable object (.po) files need to be compiled into the gettext machine object (.mo) files before translations will appear. This is done once during initial setup and provisioning, but will be out of date when the kuma locales are updated. To refresh the translations, first update the submodule in your host system:

32 Chapter 2. Getting started Kuma Documentation, Release latest

git submodule update --init --depth=10 locale

Next, enter the development environment, then: 1. Compile the .po ﬁles: make localecompile

2. Update the static JavaScript translation catalogs: make compilejsi18n

3. Collect the built ﬁles so they are served with requests: make collectstatic

2.10.2 Update the localizations in Kuma

To get the latest localization from Pontoon users, you need to update the submodule.

Note: This task is done by MDN staff or by automated tools during the push to production. You should not do this as part of a code-based Pull Request.

On the host system: 1. Create a new branch from kuma master:

git remote -v | grep origin # Should be main kuma repo git fetch origin git checkout origin/master git checkout -b update-locales # Pick your own name. Can be combined # with updating the kumascript submodule.

2. Update the locale submodule to master: cd locale git fetch git checkout origin/master cd ..

3. Commit the update: git commit locale -m "Updating localizations"

4. Push to GitHub (your fork or main repository), and open a Pull Request. It is possible to break deployments by adding a bad translation. The TravisCI job TOXENV=locales will test that the deployment should pass, and should pass before merging the PR.

2.10.3 Update the localizable strings in Pontoon

When localizable strings are added, changed, or removed in the code, they need to be gathered into .po ﬁles for translation. The TravisCI job TOXENV=locales attempts to detect when strings change by displaying the differences in locale/templates/LC_MESSAGES/django.pot, but only when a msgid changes. When this happens, the strings need to be exported to the mdn-l10n repository so that they are available in Pontoon. If done incorrectly, then the work of localizers can be lost. This task is done by staff when preparing for deployment.

2.10. Localization 33 Kuma Documentation, Release latest

2.10.4 Add a new locale to Pontoon

The process for getting a new locale on MDN is documented at Starting a new MDN localization. One step is to enable translation of the UI strings. This will also enable the locale in development environments and on https://developer.allizom.org.

Note: This task is done by MDN staff.

This example shows adding a Bulgarian (bg) locale. Change bg to the locale code of the language you are adding. 1. Updating the localizable strings in Pontoon as above, so that your commit will be limited to the new locale. 2. In kuma/settings/common.py, add the locale to ACCEPTED_LOCALES and CANDIDATE_LOCALES, and increase PUENTE[’VERSION’]. 3. Download the latest languages.json from https://product-details.mozilla.org/1.0/languages.json and place it at kuma/settings/languages.json. 4. Add the locale to translate_locales.html and the locale/ folder: make locale LOCALE=bg

5. Generate the compiled ﬁles for all the locales, including the new one: make localerefresh

6. Restart the web server and verify that Django loads the new locale without errors by visiting the locale’s home page, for example http://localhost:8000/bg/. 7. Commit the locale submodule and push to mdn-l10n, as described above in Updating the localizable strings in Pontoon. The other locales should include a new string representing the new language. 8. (Optional) Generate migrations that includes the new locale: ./manage.py makemigrations users wiki --name update_locale

9. Commit the changes to locale, jinja2/includes/translate_locales.html, and kuma/settings, and open a Pull Request. 10. Enable the language in Pontoon, and notify the language community to start UI translations.

2.10.5 Enable a new locale on MDN

Once the new translation community has completed the rest of the process for starting a new MDN localization, it is time to enable the language for page translations:

Note: This task is done by MDN staff.

1. Remove the locale from CANDIDATE_LOCALES in kuma/settings/common.py. Ensure it remains in ACCEPTED_LOCALES. 2. Restart the web server and verify that Django loads the new locale without errors by visiting the locale’s home page, for example http://localhost:8000/bg/. 3. Commit the change to kuma/settings/common.py and open a Pull Request. When the change is merged and deployed, inform the localization lead and the community that they can begin trans- lating content.

34 Chapter 2. Getting started Kuma Documentation, Release latest

2.11 CKEditor

MDN Web Docs uses a WYSIWYG editor called CKEditor. CKEditor is an open source utility which brings the power of rich text editing to the web. This document details how to update CKEditor within the MDN codebase.

2.11.1 Building CKEditor

To rebuild CKEditor, run this on the host system: cd kuma/static/js/libs/ckeditor/ ./docker-build.sh # Creates a Java container, builds CKEditor docker-compose exec web make build-static

This builds CKEditor within a Docker VM, using the Java ckbuilder tool from CKSource, then builds static resources so that the updated editor is installed where it belongs. To rebuild CKEditor locally, if you have Java installed: cd kuma/static/js/libs/ckeditor/source/ ./build.sh docker-compose exec web make build-static

Portions of the build process will take a few minutes so don’t expect an immediate result.

2.11.2 Updating CKEditor

To update the CKEditor version, you’ll need to edit build.sh and change the value of CKEDITOR_VERSION. Updating is important to keep MDN in sync with CKEditor’s functional and security updates.

2.11.3 Updating CKEditor plugins

Some plugins are maintained by MDN staff in the Kuma repo. Others are updated to the tag or commit number speciﬁed in build.sh: • descriptionlist • scayt • wsc • wordcount Change to CKEditor plugins which are bundled into Kuma should be made in the directory /kuma/static/js/libs/ckeditor/source/ckeditor/plugins/. Once you’ve made changes to a plugin, be sure to build the editor and the static resources again, as described in Building CKEditor.

2.11.4 Committing Changes

When updating CKEditor, adding a plugin, or changing the configuration, CKEditor should be rebuilt, and the results added to a new pull request. We currently do not check-in CKEditor source, but do add third-party plugin sources. We check in the “built” files, which combine CKEditor with plugins and configuration, so that they do not need to be rebuilt by the static files process. This is enforced by .gitignore, so git add can be used:

2.11. CKEditor 35 Kuma Documentation, Release latest

git add kuma/static/js/libs/ckeditor/

Reviewers should rebuild CKEditor to ensure this was done correctly. Building is not atomic, because a hex-encoded timestamp is embedded in some miniﬁed ﬁles: kuma/static/js/libs/ckeditor/build/ckeditor/ckeditor.js skins/moono/editor.css skins/moono/editor_*.css

Small changes (with big diffs) to these ﬁles are expected. Changes to other ﬁles are not expected.

2.12 Getting Data

A Kuma instance without data is useless. You can view the front page, but almost all interactive testing requires users, wiki documents, and other data.

2.12.1 The Sample Database

The sample database has a minimal set of data, based on production data, that is useful for manual and automated testing. This includes: • All documents linked from the English homepage, including: – Translations – Some historical revisions – Minimal author profiles for revisions • Additional documents for automated testing and feature coverage • Good defaults for contance variables KUMASCRIPT_TIMEOUT and KUMASCRIPT_MAX_AGE • Waffle flags and switches • Search filters and tags (but not a search index, which must be created locally - see Indexing documents) • The Mozilla Hacks feed • Test users, with appropriate groups and permissions: – test-super - A user with full permissions – test-moderator - A staff content moderator – test-new - A regular user account – test-banned - A banned user – viagra-test-123 - An unbanned spammer Test accounts can be accessed by entering the password test-password in the Django admin, and then returning to the site. See Load the sample database for instructions on loading the latest sample database.

36 Chapter 2. Getting started Kuma Documentation, Release latest

2.12.2 Add an MDN User

If you need a user proﬁle that is not in the sample database, you can scrape it from production or another Kuma instance, using the scrape_user command. In the container (after make bash or similar), run the following, replacing username with the desired user’s username: ./manage.py scrape_user username ./manage.py scrape_user https://developer.mozilla.org/en-US/profiles/username

Some useful options: --email [email protected] Set the email for the user, which can’t be scraped from the proﬁle. With the correct email, the user’s proﬁle image will be available. --social Scrape social data for the user, which is not scraped by default --force If a user exists in the current database, update it with scraped data. For full options, see ./manage.py scrape_user --help A local user can be promoted to staff with the command: ./manage.py ihavepower username --password=password

2.12.3 Add an MDN Wiki Document

If you need a wiki page that is not in the sample database, you can scrape it from production or another Kuma instance, using the scrape_document command. In the container (after make bash or similar), run the following, using the desired URL: ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS/display

Scraping a document includes: • The parent documents (such as Web and Web/CSS for Web/CSS/display) • A revision for each document, and the author for each revision • The English variant, if a translation is requested • Redirects, if a page move is detected It does not include: • Attachment data, or the attachments themselves. These will continue to use production URLs. Some useful options: --revisions REVS Scrape more than one revision --translations Scrape the translations of a page as well --depth DEPTH Scrape one or more levels of child pages as well --force Refresh an existing Document, instead of skipping For full options, see ./manage.py scrape_document --help

2.12. Getting Data 37 Kuma Documentation, Release latest

2.12.4 Add Documents Linked from a Page

If you need all the documents linked from a page in production or another Kuma instance, you can use the scrape_links command. In the container (after make bash or similar), run the following, using the desired URL: ./manage.py scrape_links # Scrape the homepage ./manage.py scrape_links https://developer.mozilla.org/en-US/Web/CSS/display

This treats a page a lot like a web crawler would, looking for wiki document links with the same locale from: • The header • The footer • The content • KumaScript-rendered sidebars and content This can result in a lot of trafﬁc. There are options that don’t affect the initial link scrape, but that are passed on to the scraped documents: --revisions REVS Scrape more than one revision --translations Scrape the translations of a page as well --depth DEPTH Scrape one or more levels of child pages as well

2.12.5 Create the Sample Database

These scraping tools are used to create a sample database of public information, which is used for development environments and functional testing without exposing any private production data. When it is time to create a new sample database, an MDN staff person runs the commamd in the the container: time scripts/create_sample_db.sh

This takes 2 to 2½ hours with a good internet connection. This is then uploaded to the mdn-downloads site: • https://mdn-downloads.s3-us-west-2.amazonaws.com/index.html • https://mdn-downloads.s3-us-west-2.amazonaws.com/mdn_sample_db.sql.gz This uses the speciﬁcation at etc/sample_db.json, which includes the sources for scraping, as well as ﬁxtures needed for a working development and testing environment.

2.12.6 Load Custom Data

The sample_mdn command does the work of creating the sample database. It can also be used with a different specification to load custom fixtures and scrape additional data for your local environment. For example, loading a new sample database wipes out existing data, so you’ll need to run the instructions in Enable GitHub authentication (optional) again. Instead, you can create a specification for your development user and GitHub OAuth application: { "sources":[ ["user", "my_username", { "social": true,

38 Chapter 2. Getting started Kuma Documentation, Release latest

"email":"[email protected]" } ] ], "fixtures":{ "users.user":[ { "username":"my_username", "email":"[email protected]", "is_staff": true, "is_superuser": true } ], "socialaccount.socialapp":[ { "name":"GitHub", "client_id":"client_id_from_github", "secret":"secret_from_github", "provider":"github", "sites":[[1]] } ], "socialaccount.socialaccount":[ { "uid":"uid_from_github", "user":["my_username"], "provider":"github" } ], "account.emailaddress":[ { "user":["my_username"], "email":"[email protected]", "verified": true } ] } }

To use this, you’ll need to replace the placeholders: • my_username - your MDN username • [email protected] - your email address, veriﬁed on GitHub • client_id_from_github - from your GitHub OAuth app • secret_from_github - from your GitHub OAuth app • uid_from_github - from your MDN SocialAccount Save it, for example as my_data.json, and, after loading the sample database, load the extra data: ./manage.py sample_mdn my_data.json

This will allow you to quickly log in again using GitHub auth after loading the sample database.

2.12. Getting Data 39 Kuma Documentation, Release latest

2.12.7 Anonymized Production Data

The production database contains conﬁdential user information, such as email addresses and authentication tokens, and it is not distributed. We try to make the sample database small but useful, and provide scripts to augment it for speciﬁc uses, reducing the need for production data. Production-scale data is occasionaly needed for development, such as testing the performance of data migrations and new algorithms, and for the staging site. In these cases, we generate an anonymized copy of production data, which deletes authentication keys and anonymizes user records. This is generated with the script scripts/clone_db.py on a recent backup of the production database. You can see a faster and less resource-intensive version of the process by running it against the sample database: scripts/clone_db.py -H mysql -u root -p kuma -i mdn_sample_db.sql.gz anon_db

This will generate a ﬁle anon_db-anon-20170606.sql.gz, where the date is today’s date. To check that the anonymize script ran correctly, load the anonymized database dump and run the check script:

zcat anon_db-anon-20170606.sql.gz | ./manage.py dbshell cat scripts/check_anonymize.sql | ./manage.py dbshell

This runs a set of counting queries that should return 0 rows. A similar process is used to anonymize a recent production database dump. The development environment is not tuned for the I/O, memory, and disk requirements, and will fail with an error. Instead, a host-installed version of MySQL is used, with the custom collation. The entire process, from getting a backup to uploading a conﬁrmed anonymized database, takes about half a day. We suspect that a clever user could de-anonymize the data in the full anonymized database, so we do not distribute it, and try to limit our own use of the database.

2.13 Documentation

This documentation is generated and published at Read the Docs whenever the master branch is updated. GitHub can render our .rst documents as ReStructuredText, which is close enough to Sphinx for most code reviews, without features like links between documents. It is occasionally necessary to generate the documentation locally. It is easiest to do this with a virtualenv on the host system, using Docker only to regenerate the MDN Sphinx template. If you are not comfortable with that style of development, it can be done entirely in Docker using docker-compose.

2.13.1 Generating documentation

Sphinx uses a Makefile in the docs subfolder to build documentation in several formats. MDN only uses the HTML format, and the generated document index is at docs/_build/html/index.html. To generate the documentation in a virtualenv on the host machine, ﬁrst install the requirements: pip install -r requirements/docs.txt

Then switch to the docs folder to use the Makefile: cd docs make html python -m webbrowser file://${PWD}/_build/html/index.html

40 Chapter 2. Getting started Kuma Documentation, Release latest

To generate the documentation with Docker: docker-compose run --rm --user $(id -u) web sh -c "\ virtualenv /tmp/.venvs/docs && \ . /tmp/.venvs/docs/bin/activate && \ pip install -r /app/requirements/docs.txt && \ cd /app/docs && \ make html" python -m webbrowser file://${PWD}/docs/_build/html/index.html

A virtualenv is required, to avoid a pip bug when changing the version of a system-installed package.

2.14 Docker

Docker is used for development and for deployment.

2.14.1 Docker Images

Docker images are used in development, usually with the local working files mounted in the images to set behaviour. Images are built by Jenkins, after tests pass, and are published to DockerHub. We try to store the configuration in the environment, so that the published images can be used in deployments by setting environment variables to deployment-specific values. Here are some of the images used in the Kuma project:

kuma

The kuma Docker image builds on the kuma_base image, installing a kuma branch and building the assets needed for running as a webservice. The environment can be customized for different deployments. The image can be recreated locally with make build-kuma. The image tagged latest is used by default for development. It can be created locally with make build-kuma VERSION=latest. The ofﬁcial latest image is created from the master branch in Jenkins and published to Docker- Hub. kuma_base

The kuma_base Docker image contains the OS and libraries (C, Python, and Node.js) that support the kuma project. The kuma image extends this by installing the kuma source and building assets needed for production. The image can be recreated locally with make build-base. The image tagged latest is used by default for development. It can be created localled with make build-base VERSION=latest. The ofﬁcial latest image is created from the master branch in Jenkins and published to Docker- Hub kumascript

The kumascript Docker image contains the kumascript rendering engine and support ﬁles. The environment can be customized for different deployments. The image can be recreated locally with make build-kumascript.

2.14. Docker 41 Kuma Documentation, Release latest

The image tagged latest is used by default for development. It can be created locally with make build-kumascript KS_VERSION=latest. The ofﬁcial latest image is created from the master branch in Jenkins and published to DockerHub. integration-tests

The integration-tests Docker image contains browser-based integration tests that check the functionality of a running Kuma deployment. The image can be recreated locally with docker build -f docker/images/integration-tests/ ., but this is only necessary for image development. Most developers will follow the Client-side testing documentation to develop and run these integration tests. The image is built and used in Jenkins in the stage-integration-tests and prod-integration-tests pipelines, conﬁgured by scripts in the Jenkinsfiles folder. It is not published to DockerHub.

2.15 Deploying MDN

MDN is served from Amazon Web Services in the US West (Oregon) datacenters. MDN is deployed as Docker containers managed by Kubernetes. The infrastructure is deﬁned within MDN’s infra repository, including the tools used to deploy and maintain MDN. Deploying new code to AWS takes several steps, and about 60 minutes for the full process. The deployer is responsible for emergency issues for the next 24 hours. There are 1-3 deployments per week, usually between 7 AM and 1 PM Paciﬁc, Monday through Thursday.

Note: This page describes deployment performed by MDN staff. It requires additional setup and permissions not described here. Deployments will not work for non-staff developers, and should not be attempted.

2.15.1 Pre-Deployment

Before deploying, a staff member should: • (Once a week) Check that the latest mdn-browser-compat-data node package is included in the KumaScript image. The latest package is installed when the image is created, during the RUN npm update step, and can be veriﬁed by reading the build logs or running the image. • (Once a week) Update the locale and kumascript submodules. The locale submodule must be updated to apply new translations to production. The kumascript submodule is not used for deployment, but updating it keeps the development environment in sync. – Commit or stash any changes in your working copy. – Create a pre-push branch from master: git fetch origin git checkout -b pre-push-`date +"%Y-%m-%d"` origin/master

– Update the submodules: git submodule update --remote git commit -m "Updating submodules" kumascript locale

42 Chapter 2. Getting started Kuma Documentation, Release latest

– Push new localizable strings. This step is only necessary when there are new strings. The TravisCI job TOXENV=locales checks for new strings, and you can examine the output of the recent master build to see if there are differences. At the same time, it’s not too hard to run locally and then revert if there are no useful changes. 1. On the host system, checkout the master locale branch:

cd locale git checkout master git pull

2. Update kuma/settings/common.py, and bump the version in PUENTE[’VERSION’]. 3. Inside the development environment, extract and rebuild the translations:

make localerefresh

4. On the host system (still in the locale subfolder), review the changes to source English strings:

git diff templates/LC_MESSAGES

5. If there are useful changes (ignore comment lines and look for msgid as the changed line), commit the ﬁles in the locale submodule:

git add --all . git commit

For the commit message, use the PUENTE[’VERSION’] in the commit subject, and summarize the string changes in the commit body, like:

Update strings 2018.08

* Add "View All" text for document history pagination

Attempt to push to the mdn-l10n repository:

git push

If this fails, do not force with –force, or attempt to pull and create a merge commit. Someone has added a translation while you were working, and you need to start over to preserve their work:

git fetch git reset --hard @{u}

This resets your locale submodule to the new master. Start over on step 3 (make localerefresh). If the push to mdn-l10n is a success, commit your Kuma changes:

cd .. git commit kuma/settings/common.py locale

You can use the same commit message used for the locale commit. 6. If there are no new strings, throw the changes away, starting in the locale folder:

2.15. Deploying MDN 43 Kuma Documentation, Release latest

git checkout -- . cd .. git checkout -- kuma/settings/common.py

– Push the branch and open a pull request: git push -u origin pre-push-`date +"%Y-%m-%d"`

– Check that tests pass. The TOXENV=locales job, which uses Dennis to check strings, is the job that occasionally fails. This is due to a string change committed in Pontoon that will break rendering in production. If this happens, ﬁx the incorrect translation, push to master on mdn-l10n, and update the submodule in the pull request. Wait for the tests to pass. – Merge the pull request. A “Rebase and merge” is preferred, to avoid a merge commit with little long-term value. Delete the pre-push branch. • Check the latest images from master are built by Jenkins(Kuma master build, KumaScript master build, both private to staff), and uploaded to the DockerHub repositories (Kuma images, KumaScript images).

2.15.2 Deploy to Staging

The staging site is located at https://developer.allizom.org. It runs on the same Kuma code as production, but against a different database, other backing services, and with less resources. It is used for verifying code changes before pushing to production. • Start the staging push, by updating and pushing the stage-push branches: git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout stage-push git merge --ff-only origin/master git push cd ..

• Prepare for testing on staging: – Look at the changes to be pushed (What’s Deployed on Kuma, and What’s deployed on KumaScript). To enlist the help of pull request authors and others, you can report bug numbers and PRs in IRC. firebot will give handy links to Bugzilla. – Think about manual tests to conﬁrm the code changes work without errors. – Monitor the push in the #mdndev IRC channel. The ﬁnal messages (one for kuma, one for kumascript) look like: SUCCESS: Check Rollout Status: Branch stage-push build #104:

• Merge and push to the stage-integration-tests branch: git checkout stage-integration-tests git merge --ff-only origin/master git push

This will kick off functional tests in Jenkins, which will also report to #mdndev.

44 Chapter 2. Getting started Kuma Documentation, Release latest

• Manually test changes on https://developer.allizom.org. Look for server errors on homepage and article pages. Try to verify features in the newly pushed code. Check the functional tests. • Announce in IRC that staging looks good, and you are pushing to production.

2.15.3 Deploy to Production

The production site is located at https://developer.mozilla.org. It is monitored by the development team and MozMEAO. • Pick a push song on https://www.youtube.com. Post link to IRC. • Start the production push: git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout prod-push git merge --ff-only origin/master git push cd ..

• Monitor the push in the #mdndev IRC channel. The ﬁnal messages (one for kuma, one for kumascript) are something like: SUCCESS: Check Rollout Status: Branch prod-push build #7

• For the next 30-60 minutes, – Watch https://developer.mozilla.org – Monitor MDN in New Relic for about an hour after the push, for increased errors or performance changes. – Start the standby environment deployment – Close bugs that are now ﬁxed by the deployment – Move relevant Taiga cards to Done – Move relevant Paper cut cards to Done

2.15.4 Deploy to Standby Environment

The standby environment is located in the AWS EU Frankfurt datacenter. It runs the same code and database as production, but runs in read-only maintenance mode and on minimal resources. It will be scaled up and handle MDN trafﬁc if there is a critical failure in the AWS US West datacenter. • Start the standby environment push: git fetch origin git checkout standby-push git merge --ff-only origin/master git push cd kumascript git fetch origin git checkout standby-push git merge --ff-only origin/master

2.15. Deploying MDN 45 Kuma Documentation, Release latest

git push cd ..

• Monitor the push in the #mdndev IRC channel. The ﬁnal messages (one for kuma, one for kumascript) are something like: SUCCESS: Check Rollout Status: Branch standby-push build #7

2.16 Rendering Documents

Kuma’s wiki format is a restricted subset of HTML, augmented with KumaScript macros {{likeThis}} that render to HTML for display. Rendering is done in several steps, many of which are stored in the database for read-time performance.

A revision goes through several rendering steps before it appears on the site: 1. A user submits new content, and Kuma stores it as Revision content 2. Kuma bleaches and ﬁlters the content to create Bleached content 3. KumaScript renders macros and returns KumaScript content 4. Kuma bleaches and ﬁlters the content again to create Rendered content

46 Chapter 2. Getting started Kuma Documentation, Release latest

5. Kuma divides and processes the content into Body HTML, Quick links HTML, ToC HTML, Summary text and HTML There are other rendered outputs: • Kuma normalizes the Revision content into the Diff format to comparing revisions • Kuma filters the Revision content and adds section IDs to create the Triaged content for updating pages • KumaScript renders the Triaged content as Preview content • Kuma filters the Bleached content and adds section IDs to publish the Raw content • Kuma extracts code sections from the Rendered content to flesh out a Live sample • Kuma extracts the text from the Rendered content for the Search content

2.16.1 Revision content

CKEditor provides a visual HTML editor for MDN writers. The raw HTML returned from CKEditor is stored in the Kuma database for further processing. source User-entered content, usually via CKEditor from the edit view (URLs ending with $edit) Developer-submitted content via an HTTP PUT to the API view (URLs ending in $api) displayed on MDN “Revision Source” section of the revision detail view (URLs ending with $revision/), in a

 tag database wiki_revision.content code kuma.wiki.models.Revision.content To illustrate rendering, consider a new document published at /en-US/docs/Sandbox/simple with this Revision content: {{ CSSRef }}
I am a simple document with a CSS sidebar.
I am red.
Some Links
 The HTML Reference
 {{HTMLElement('div')}}
 A new document
 
This document has elements that highlight different areas of rendering: • A sidebar macro CSSRef, which will be rendered by KumaScript and extracted for display •A 
 tag, which will gain an id attribute • A list of three links: 1. An HTML link to an existing document
2.16. Rendering Documents 47 Kuma Documentation, Release latest
2. A reference macro HTMLElement which will be rendered by KumaScript 3. An HTML link to a new document, which will get rel="nofollow" and class="new" at- tributes • An onclick attribute, added in Source mode, which will be removed •A   
The Revision content is normalized using pytidylib, a Python interface to the C tidylib library, which turns the content into a well-structured HTML 4.01 document. Content difference reports, or “diffs”, are generated by a line-by-line comparison of the content in Diff format of two revisions. Lines that differ are dropped, so that the reports focus on just the changed content, often without the wrapping HTML tags like . These diffs often contain line numbers from the Diff format, which do not correspond to the line numbers in the Revision content because of differences in formatting and whitespace. Because the Diff format can contain unsafe content, it is not displayed directly on MDN. On Revision comparison views, the Revision dashboard, and in feeds, two Diff formats are processed by difﬂib.HtmlDiff to generate an HTML 
 showing only the changed lines, and with HTML escaping for the content. For emails, difﬂib.uniﬁed_diff generates a text-based difference report, and it is sent as a plain-text email without escaping.
2.16.10 Triaged content
When a document is re-edited, the Revision content of the current revision is processed before being sent to the editor. This is a lighter version of the full bleaching process used to create Bleached content and Rendered content. source Revision content, with further processing in RevisionForm. displayed on MDN Editing  in the edit view (URLs ending with $edit) Editing <textarea in the translate view (URLs ending with $translate) database not stored code not available For the simple document, this is the Triaged content: <p>{{ CSSRef }}</p></p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p>54 Chapter 2. Getting started Kuma Documentation, Release latest</p><p><p style="color:red">I am red.</p></p><p><h2 id="Some_Links">Some Links</h2></p><p><ul> <li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li>{{HTMLElement('div')}}</li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p><script> alert('How about this?'); </script></p><p>The headers get IDs, based on the content, if they did not have them before. For example, id="Some_Links" is added to the <h2>. A simple ﬁlter is applied that strips any attributes that start with on, such as the scripting attempt onclick. Further bleaching, for example to remove the <script>, is not applied. CKEditor will perform additional parsing and formatting at load time. It will sometimes notice the empty <div> and replace it with <div class="button"> </div>, especially if it is the last element on the page. It may also remove the <script> element entirely. If a writer makes a change, these backend and CKEditor changes will be reﬂected in the new Revision content. This can confuse writers (“I didn’t add those IDs!”).</p><p>2.16.11 Preview content</p><p>When editing, a user can request a preview of the document. This sends the in-progress document to editing, with a smaller list of environment variables. source Triaged content, with CKEditor parsing, passed through KumaScript output HTML content at /<locale>/docs/preview-wiki-content database not stored code not available The Preview content for the simple document is: </p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p><p style="color: red;">I am red.</p></p><p><h2>Some Links</h2></p><p><ul> <li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li><a href="/en-US/docs/Web/HTML/Element/div" title="The HTML Content Division element (<div>) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."><code><div></code></a></li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p>2.16. Rendering Documents 55 Kuma Documentation, Release latest</p><p><script> alert('How about this?'); </script></p><p>Fewer environment variables are passed to the KumaScript server for preview than when generating the KumaScript content: url The base URL of the website, like https://developer.mozilla.org/ locale The locale of the request, like "en-US" Some macros use the absence of an environment variable to detect preview mode, and change their output. For example, {{CSSRef}} notices that env.slug is not deﬁned, and outputs an empty string, leaving in the preview output. Other macros don’t have speciﬁc code to detect preview mode, and have KumaScript rendering errors in preview. Some macros, like {{HTMLElement}}, work as expected in preview.</p><p>2.16.12 Raw content</p><p>A ?raw parameter can be added to the end of a document to request the source for a revision. This is processed in a similar way to the Triaged content, but from the Bleached content. source Bleached content, with ﬁlters output The page with a ?raw query parameter database not stored code not available For the simple document, this is the raw content: <p>{{ CSSRef }}</p></p><p><p>I am a <strong>simple document</strong> with a CSS sidebar.</p></p><p><p style="color: red;">I am red.</p></p><p><h2 id="Some_Links">Some Links</h2></p><p><ul> <li><a href="/en-US/docs/Web/HTML">The HTML Reference</a></li> <li>{{HTMLElement('div')}}</li> <li><a href="/en-US/docs/NewDocument">A new document</a></li> </ul></p><p><div class="button"></div></p><p><script> alert('How about this?'); </script></p><p>The Bleached content is parsed for ﬁltering . The headers get IDs, based on the content, if they did not have them before. For example, id="Some_Links" is added to the <h2>. A simple ﬁlter is applied that strips any attributes that start with on, such as the scripting attempt onclick. However, this step should do nothing, since these attribute are dropped when creating the Bleached content.</p><p>56 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>2.16.13 Live sample</p><p>Live samples are stored in document content. The content is then processed to extract the CSS, JS, and HTML, and reformat them as a stand-alone HTML document suitable for displaying in an <iframe>. source A section extracted from Rendered content, with further processing output Live sample documents on a separate domain, such as https://mdn.mozillademos.org database Not stored in the database, but cached code kuma.wiki.Document.extract.code_sample(section_id) The simple document does not include one of these samples. The Live samples page on MDN describes how the system works for content authors, and includes a live sample demo. Most live samples are loaded in an <iframe>, inserted by the macro EmbedLiveSample. If the sample doesn’t work as an <iframe>, LiveSampleLink can be used instead. The <iframe src= URL is Kuma, running on a different domain, such as https://mdn.mozillademos.org, and conﬁgured to serve live samples (the code sample view) and attachments. A separate domain for user-created content, often served in an <iframe>, mitigates many security issues. The live sample is cached on ﬁrst access, and generated when requested. The extractor looks for <pre> sections with class="brush: html", "brush: css", and "brush: js", to ﬁnd the sample content, and then selectively un-escapes some HTML and CSS. These sections are used to populate a basic HTML ﬁle. There are other sample types that are not derived from wiki content. These are out-of-scope for this document, but the most signiﬁcant are listed here for the curious: • Legacy samples, like cssref/background-attachment.html, are no longer maintained and are planned for removal (see bug 1076893 and related bugs). • GitHub Live Samples, like the CSS circle demo, are maintained in an MDN repo like mdn/css-examples, served by GitHub pages, and inserted with EmbedGHLiveSample. • Interactive examples are sourced in the mdn/interactive-examples repository, deployed as a static website, inserted with the EmbedInteractiveExamples macro near the top of the page, and are displayed in an <iframe>.</p><p>2.16.14 Search content</p><p>Wiki documents are converted to a JSON format and indexed by ElasticSearch for internal search. This allows search- ing for words in the wiki content. source text extracted from Rendered content output text sections from in-content results from internal search database stored in ElasticSearch code kuma.wiki.search.WikiDocumentType.from_django(Document) The Django utility strip_tags is used to quickly remove HTML tags. This utility is not guarenteed to generate an HTML safe string, as highlighted in a security advisory. Kuma does not redisplay this string. ElasticSearch applies the HTML Strip Char Filter to this and other content, which also strips tags and replaces HTML entities like & with the character equivalents like &. When a search result is picked because of a content match, ElasticSearch returns the matching section, highlighting the matching terms in bold. This HTML is redisplayed on the search results page. Documents are indexed when created and when updated, as an asyncronous process. Documented are removed from the index when deleted. Administrators can also re-create the entire index, for ElasticSearch upgrades or to freshen the data.</p><p>2.16. Rendering Documents 57 Kuma Documentation, Release latest</p><p>There is additional page metadata sent to ElasticSearch to power internal search. This includes page titles, tags, and locales. It also includes KumaScript macros, CSS class names, and HTML attributes, to allow advanced search queries and to power the macro dashboard.</p><p>2.16.15 Future Changes</p><p>Rendering evolved over years, and this document describes how it works, rather than how it was designed. There are some potential changes that would simplify rendering: • Sidebar macros are heavy users of API data and require post-processing of the Rendered content. Sidebar generation could be moved into Kuma instead of being speciﬁed by a macro. • The Diff format could be replaced by the Bleached content format, which would be stored for each revision rather than just for the most recent document. • Content from editing could be normalized and ﬁltered before storing as the Revision content. This may unify the Triaged content, Diff format, and Bleached content. • The views that accept new revisions could add IDs to the content before storing the Revision content, rather than wait for the Triaged content or Body HTML. • Developers could refactor the code to consistently access and generate content, rather than repeat ﬁlter logic in different forms and views.</p><p>2.16.16 History</p><p>MDN has used different rendering processes in the past. Prior to 2004, Netscape’s DevEdge was a statically-generated website, with content stored in a revision control system (CVS or similar). This was shut down for a while, until Mozilla was able to negotiate a license for the content. From 2005 to 2008, MediaWiki was used as the engine of Mozilla Developer Center. The DevEdge content was converted to MediaWiki Markup. From 2008 to 2011, MindTouch DekiWiki was used as the engine. MindTouch migrated the MediaWiki content to the DekiWiki format, a restricted subset of HTML, augmented with macros (“DekiScript”). During this period, the site was rebranded as Mozilla Developer Network. In 2011, Kuma was forked from Kitsune, the Django-based platform for support.mozilla.org. The wiki format was as close as possible to the DekiWiki format. A new service KumaScript was added to implement DekiScript-style macros. The macros, also known as templates, were stored as content in the database. The service had a GET API to render pages, and a POST API to render previews. In 2013, content zones were added, which allowed a “zone” of pages to have a different style from the rest of the site. For example, the Firefox Zone of all the documents under /Mozilla/Firefox had a logo and a shared sub-navigation sidebar. Sub-navigation was similar to quick links, identiﬁed by <section id="Subnav">, but stored on the “zone root” (/Mozilla/Firefox) rather than generated by a macro. Zones were part of an effort to consolidate developer documentation on MDN. In 2016, the macros were exported from the Kuma database into the macros folder in the KumaScript repository. The historical changes were exported to mdn/archived_kumascript. This made rendering faster, and allowed code reviews and automated tests of macros, at the cost of requiring review and a production push to deploy macro changes. In 2018, the content zones feature was removed. This was part of an effort to focus MDN Web Docs on common web platform technologies, and away from Mozilla-speciﬁc documentation. The sub-navigation feature was dropped. In 2019, the KumaScript engine and macros were modernized to use current features of JavaScript, such as async / await, rather than libraries common in 2011. The API was also uniﬁed, so that both previews and standard renders required a POST.</p><p>58 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>2.17 Building, collecting, and serving assets</p><p>MDN requires static assets (images, JavaScript, CSS, and other ﬁles) to augment the HTML generated by Django. Kuma uses a combination of technologies and techniques to build and deliver static assets. Many of the build and development processes are documented on the Installation and Development documents. This document goes into the details. The three phases of an asset’s life are Building, Collecting, and Serving. • Building - Source ﬁles are processed to build intermediate and ﬁnal assets – Locale ﬁles contain translatable strings extracted from source ﬁles, are translated by humans, and are compiled to a binary representation for runtime use. – Localization JavaScript ﬁles contain translated strings in an executable format to allow in-browser translation of UI strings. – CKEditor is packaged with custom plugins for MDN’s use cases. – Many JavaScript libraries that are used on MDN are included in Kuma’s repository for version con- trol. – JavaScript bundles, assembled by django-pipeline, combine several source ﬁles into a single miniﬁed JavaScript ﬁle. – CSS bundles, assembled by django-pipeline or gulp, combine several source Sass ﬁles into a single miniﬁed CSS ﬁle. • Collecting - Built assets are collected to the /static folder. – Django’s staticﬁles provides a framework of Finder and Storage classes for collecting ﬁles. – django-pipeline augments staticfiles, to allow compiling, bundling, and miniﬁying JS and CSS assets in the collection phase. – gulp provides alternative tools to staticfiles for local development of assets. • Serving - Collected assets are served to visitors in development and production – In Django and Jinja templates, the static tag returns the URL of assets collected by staticfiles. – The statici18n tag returns the URL of localization JavaScript collected by staticfiles. – The pipeline tags javascript and stylesheet return the HTML markup for assets processed and collected by django-pipeline. – WhiteNoise serves static assets as part of the Django process.</p><p>2.17.1 Extracting and building locale ﬁles</p><p>Kuma uses Pontoon to translate strings in the user interface, in error messages, and in emails. These are stored in the mdn-l10n repository, and included as a git submodule at locale/. See the localization document for more details about locales. Puente extracts strings to the Portable Object Template (.pot) ﬁles, as speciﬁed in the PUENTE conﬁguration. The ﬁle locale/templates/LC_MESSAGES/django.pot contains strings from template ﬁles and Python code. The ﬁle javascript.pot contains strings from JavaScript ﬁles. Puente looks for the string parameters of gettext functions, such as gettext(), the common alias _(), and ngettext(). It also parses longer strings in the template tag trans. Next the changes are merged into the existing Portable Object (.po) ﬁles, such as locale/fr/LC_MESSAGES/django.po, to add new strings and comment out removed strings.</p><p>2.17. Building, collecting, and serving assets 59 Kuma Documentation, Release latest</p><p>Extracting and merging is done with make localeextract, usually during deployment, when UI strings change. This uses the extract management command provided by Puente, which uses Babel to extract strings and update the catalog. A maintainer pushes the updated catalogs as a new commit to the mdn-l10n repository. Pontoon detects that the repository has changed, and notiﬁes localization teams that there are new strings. In about 48 hours, the most active teams will translate strings into the top 10 MDN languages. These are applied by updating the locale submodule during the deployment process. At run time, Machine Object (.mo) ﬁles, such as locale/fr/LC_MESSAGES/django.mo, are used by gettext functions, like gettext() and _(), to display the localized strings. These are built with make localecompile when creating the production images or when a developer wants to see updated translations.</p><p>2.17.2 Building localization JavaScript</p><p>Django includes a JavaScriptCatalog view that provides JavaScript implementations of gettext functions, as well as translations for each locale. It is ineffecient to use this view directly, since it is generated on access. For efﬁciency, django-statici18n generates ﬁles for each locale from the JavaScriptCatalog output, so they can be served as static assets. The translation catalog ﬁles are created with make compilejsi18n from the locale Machine Object .mo ﬁles. This calls the compilejsi18n management command provided by django-statici18n. Kuma sets STATICI18N_ROOT to build/locale, and the output ﬁles have names like build/locale/jsi18n/de/javascript.js.</p><p>2.17.3 Building CKEditor</p><p>CKEditor is a complex JavaScript application that provides a WYSIWYG editor for MDN wiki pages. It is packaged with plugins, some from third parties, and some custom to MDN. The CKEditor build process is documented on the CKEditor document. The built ﬁles are checked into the Kuma repository.</p><p>2.17.4 Including JS libraries</p><p>Third-party JavaScript libraries are included in the Kuma repository, to avoid ambiguity about what versions of li- braries are used. Some libraries were added manually, and others with Bower. See Front-end asset dependencies for more details about these libraries. Some of these libraries are served directly to visitors, while others are included in pipleline JavaScript bundles.</p><p>2.17.5 Building pipeline JavaScript bundles</p><p>Pipeline JavaScript bundles combine several JavaScript ﬁles into a single ﬁle, with optional minimization. For exam- ple, the ﬁle static/build/js/main.js is the combination of 10 JavaScript ﬁles: • kuma/static/js/libs/jquery/jquery.js (JQuery 2.2.0) • kuma/static/js/libs/icons.js • kuma/static/js/components.js • kuma/static/js/analytics.js • kuma/static/js/main.js • kuma/static/js/components/nav-main-search.js</p><p>60 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>• kuma/static/js/auth.js • kuma/static/js/highlight.js • kuma/static/js/wiki-compat-trigger.js • kuma/static/js/lang-switcher.js The JS bundles are speciﬁed in PIPELINE_JS in the Django settings. The bundles are served differently in “de- velopment” and “production” modes. This is roughly controlled by the Django setting DEBUG, which sets further parameters like PIPELINE[PIPEINE_ENABLED], and the environment setting DJANGO_SETTINGS_MODULE, which switches the Django settings ﬁle. See django-pipeline as well as the pipeline tags section for details. In development, the source ﬁles (10 for main.js) are served, so there are 10 <script> elements in the HTML when {{javascript(’main’)}} is used in a template. In production, the output bundle is used, so a single <script> tag appears in the HTML. The single bundle is also processed with UglifyJS, which removes whitespace, replaces variable names with shorter names, and performs other transformations to make the ﬁle smaller.</p><p>2.17.6 Building pipeline CSS bundles</p><p>Pipeline CSS bundles are conceptually similar to Pipeline JS Bundles. Some contain multiple source ﬁles, such as static/build/styles/dashboards.css, which combines: • kuma/static/styles/dashboards.scss • kuma/static/styles/diff.scss Source styles are written in Sass, and compiled to CSS with node-sass. These must be compiled to CSS in both development and production modes. Backend developers tend to use make build-static to build and collect these ﬁles, and front-end developers tend to use gulp watch to directly compile them. See Front-end development for more information. The CSS bundles are speciﬁed in PIPELINE_CSS in the Django settings. The bundles are served differently in “development” and “production” modes. This is roughly controlled by the Django setting DEBUG, which sets further parameters like PIPELINE[PIPEINE_ENABLED], and the environment setting DJANGO_SETTINGS_MODULE, which switches the Django settings ﬁle. See django-pipeline as well as the pipeline tags section for details. In development, the source ﬁles (2 for dashboards.css) are used, so there are 2 <link> elements in the HTML when when {{stylesheet(’dashboards’)}} is used in a template. In production, the output bundle is used, so a single <link> tag appears in the HTML. When bundled, CSS is also processed by clean-css, which transforms the CSS to make the output ﬁles smaller.</p><p>2.17.7 Collecting asset ﬁles with staticﬁles</p><p>Django provides the django.contrib.staticﬁles app, widely used in Django projects to standardize where assets are stored, to collect them for development and production, and to use different asset URLs in different environments. In development mode, the staticfiles app helps identify assets spread across the project, and often allows a rapid development cycle (for example, change a ﬁle, refresh the browser, and see the effects of the changed ﬁle). For production, the staticfiles app provides the management command collectstatic, which gathers ﬁles to the /static folder for efﬁcent ﬁle serving. The Django documents for staticfiles are mostly focused on usage. Additional details are needed to understand how django-pipeline customizes staticfiles.</p><p>2.17. Building, collecting, and serving assets 61 Kuma Documentation, Release latest</p><p>Conﬁguration</p><p>The staticfiles app is conﬁgured by Django settings: STATIC_ROOT The folder on the ﬁle system where assets are collected. For MDN, this is the static folder in the kuma directory. STATIC_URL The base URL for static assets. In development, this is http://localhost:8000/static/, and in production it is https://developer.mozilla.org/static/. STATICFILES_FINDERS The dotted path to classes implementing staticfiles Finder. These determine what ﬁles will be collected and served. Kuma uses four ﬁnders: • django.contrib.staticﬁles.ﬁnders.FileSystemFinder: Finds ﬁles in folders speciﬁed by STATICFILES_DIRS • django.contrib.staticﬁles.ﬁnders.AppDirectoriesFinder: Finds ﬁles in the static subfolder of any in- stalled apps • pipeline.ﬁnders.CachedFileFinder: Strips hashes from ﬁlenames to identify the “pre-cached” names for ﬁles. • pipeline.ﬁnders.PipelineFinder: When combined assets are not enabled (PIPELINE[’PIPELINE_ENABLED’] == False), returns the source ﬁles instead of the combined bundle ﬁle. STATICFILES_DIRS A list of folders in the kuma directory that the FileSystemFinder will scan for static assets. For MDN, this includes: • assets/static • kuma/static • kuma/javascript/dist • build/locale • jinja2/includes/icons For example, the localization JavaScript build/locale/jsi18n/fr/javascript.js will be collected to static/jsi18n/fr/javascript.js. STATICFILES_STORAGE The dotted path to a class implementing staticfiles Storage. Storage determines where ﬁles are stored, what URLs they have, and provides hooks for modifying ﬁles when copying them. Kuma uses three different storages, depending on the context: • Development server (DEBUG=True): pipeline.storage.NonPackagingPipelineStorage, which avoids com- bining ﬁles when collecting them. • Production server (DEBUG=False): kuma.core.pipeline.storage.ManifestPipelineStorage, which com- bines packaged ﬁles, hashes the names, and creates a manifest. • Testing (pytest, TravisCI, etc.) and make commands: pipeline.storage.PipelineStorage, which com- bines packaged ﬁles but does not hash the names.</p><p>Finder classes</p><p>The staticfiles app uses Finders to locate asset ﬁles. Django considers this a private API, so it may change in the future. There are two methods the BaseFinder class expects to be implemented: • find(path): Given a short path like css/wiki.css, return the absolute path to the ﬁle. This is used by the findstatic management command, and to ﬁnd ﬁles when serving assets in development mode.</p><p>62 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>• list(ignore_patterns): Return a list of the ﬁles this Finder can ﬁnd, along with a storage instance for each. The collectstatic management command uses this to gather ﬁles. The staticfiles app provides two ﬁnders used by Kuma: • The FileSystemFinder collects ﬁles under the folders speciﬁed in the STATICFILES_DIRS setting. • The AppDirectoriesFinder collects ﬁles in the (optional) static subfolder of any installed app listed in INSTALLED_APPS. This is how Django applications, including ones bundled with Django, distribute JavaScript, CSS, images, and other assets. It isn’t used for Kuma’s apps. Instead, we’ve standardized on kuma/static and other named paths. The Finders are used by WhiteNoise to determine which ﬁle to serve in development mode. The management com- mand findstatic can be used to determine which ﬁle is served, such as: $ ./manage.py findstatic -v2 js/main.js</p><p>Found 'js/main.js' here: /app/kuma/static/js/main.js /app/static/js/main.js Looking in the following locations: /app/kuma/static /app/build/locale /app/jinja2/includes/icons /usr/local/lib/python2.7/site-packages/flat/static /usr/local/lib/python2.7/site-packages/django/contrib/admin/static /usr/local/lib/python2.7/site-packages/constance/static /usr/local/lib/python2.7/site-packages/djcelery/static /usr/local/lib/python2.7/site-packages/django_extensions/static /usr/local/lib/python2.7/site-packages/rest_framework/static /usr/local/lib/python2.7/site-packages/debug_toolbar/static /app/static</p><p>When multiple ﬁles are found, the ﬁrst is used. In the above example, /app/kuma/static/js/main.js will be served in development for /static/js/main.js.</p><p>Storage classes</p><p>The staticfiles app uses a Storage class, which extends Django’s Storage class for asset workﬂows. Django documents how to write a custom storage system, and there are many 3rd-party storage packages for using various cloud providers for ﬁle hosting. The conﬁgured STATICFILES_STORAGE class is used when collecting ﬁles with ./manage.py collectstatic. Django’s standard Storage classes provide methods like delete(), exists(), and size() for implementing ﬁle methods, and methods like listdir() for getting lists of ﬁles. There is a wide variety of storage backends with different capabilities, and Django allows most methods to raise NotImplementedErrror if an operation is not supported or is too expensive. A staticfiles Storage class extends the standard Storage classes and requires a few more methods, although the exact methods are undocumented. Some are path(name), to turn a relative path to a full path, and url(path), to get the external URL of the ﬁle. An optional method, post_process(), can be deﬁned to further process the ﬁles, and returns a map of the old paths to the new paths. The default storage, StaticFilesStorage, is based on the standard FileSystemStorage, and copies static ﬁles to STATIC_ROOT (the static folder). For the url() method, it prepends the STATIC_URL to the path. ManifestStaticFilesStorage implements the post_process() method to add the MD5 hash of the ﬁle’s contents to the ﬁlename. This allows these ﬁles to be served with very long cache times, since changes will also change the ﬁlename. It also requires manipulating the contents so that references to assets within other ﬁles, such as a</p><p>2.17. Building, collecting, and serving assets 63 Kuma Documentation, Release latest</p><p>CSS @import statement, are updated to the hashed names. This often requires source ﬁles use relative paths like ../img/logo.svg, so that the tool can ﬁnd the destination ﬁle. Because of the intense ﬁle processing, ManifestStaticFilesStorage doesn’t support the live updates of de- velopment mode. It requires DEBUG=False, and that ./manage.py collectstatic is run before running the server, or before a server restart. A map of original to hashed names is stored in staticﬁles.json, and is read at server startup to determine the hashed names. CachedStaticFilesStorage is similar to ManifestStaticFilesStorage, but stores the ﬁlename mapping in the cache. It is slower than staticfiles.json, and is used when write access to the ﬁlesystem is forbidden.</p><p>2.17.8 django-pipeline</p><p>The django-pipeline library is used for packing assets. It provides CSS and JavaScript concatenation and com- pression, built-in JavaScript template support, and optional data-URI image and font embedding. It does this by extending and overriding the django-staticﬁles app, so that assets are processed with the standard ./manage.py collectstatic command. Kuma uses django-pipeline to: • Compile Sass .sccs ﬁles plain CSS with node-sass • Combine multiple JS and CSS ﬁles into a single ﬁle (“bundle”) in production • Compress CSS ﬁles with cleancss • Compress JS ﬁles with UglifyJS</p><p>Conﬁguration</p><p>The django-pipeline app is conﬁgured with the dictionary PIPELINE. There are many conﬁguration items, some of which are: • PIPELINE_ENABLED: True to concatenate and compress assets (testing and production), and False to skip concatenation and compression. • PIPELINE_COLLECTOR_ENABLED: True to collect assets (testing and production), and False to skip collection and leave them in the source locations. • COMPILERS: A list of CSS compilers. pipeline‘s SASSCompiler in testing and production, and kuma.core.pipeline.sass.DebugSassCompiler (which does nothing, but instead defers to gulp) in development. The Makefile speciﬁes the testing conﬁguration, so commands like make collectstatic run with PIPELINE_ENABLED and PIPELINE_COLLECTOR_ENABLED. However, they are disabled when running the development server. django-pipeline speciﬁes outputs as a “package”, which speciﬁes one or more inputs, one output, and some optional settings and overrides. PIPELINE[’JAVASCRIPT’] speciﬁes the JavaScript packages, and PIPELINE[’STYLESHEETS’] speciﬁes the Sass/CSS packages.</p><p>Finders</p><p>Kuma uses two Finders from django-pipeline. CachedFileFinder strips hashes from ﬁlenames to iden- tify the “pre-cached” names for ﬁles, by removing the middle element of ﬁlenames with three dots. This may have been useful in django-pipeline 1.3 or earlier, but it appears to do nothing now, or could potentially do the wrong thing such as resolving bootstrap.min.js as bootstrap.js. PipelineFinder does nothing if PIPELINE[’PIPELINE_ENABLED’] if True (testing and production), and uses the Storage to ﬁnd ﬁles if it is</p><p>64 Chapter 2. Getting started Kuma Documentation, Release latest</p><p> disabled. For Kuma, this means it may ﬁnd ﬁles in the STATIC_ROOT directory. However, since the FileSystemFinder ﬁnds most ﬁles in kuma/static ﬁrst, it is doubtful if this Finder ever applies.</p><p>Storage</p><p>Most of the functionality of django-pipeline is implemented as a Storage class, and Kuma uses three different implementations depending on the environment. The simplest storage, used during testing and in the Makefile, is pipeline.storage.PipelineStorage, which extends the staticﬁles Storage class StaticFilesStorage, with a post_process step that packages JS and CSS into one-ﬁle bundles, according to the PIPELINE conﬁguration. Development uses pipeline.storage.NonPackagingPipelineStorage. This works the same way as PipelineStorage, but avoids creating packages, where several ﬁles are combined into one. JavaScript ﬁles are served from the source folders, but CSS ﬁles need to be com- piled from Sass, and are served from the /static folder after collection. When developing style ﬁles, a developer either needs to run ./manage.py collectstatic or use gulp to see changes. In production, kuma.core.pipeline.storage.ManifestPipelineStorage is used. This com- bines the package processing of PipelineStorage with the hashed assets and staticfiles.json of ManifestStaticFilesStorage. These are generated when the production Docker containers are created.</p><p>2.17.9 Compiling and collecting assets with Gulp</p><p>An alternate way to compile and collect assets is to use Gulp, as described in Compiling on the host system with gulp. This requires installing node and related packages on the “host” system, rather than relying on the Docker containers, but it matches the preferred workﬂow of some front-end developers. The gulp process also compiles Sass sources to CSS, and copies ﬁles from /kuma/static to /static, mir- roring the process from make collectstatic. However, additional tools, like PostCSS, can’t be added to the gulp workﬂow like other projects, because the make collectstatic process is the only one used to generate production assets.</p><p>2.17.10 Template tag static</p><p>Django provides a template tag static that outputs the URL of the static asset for HTML. Without staticfiles installed, it just adds STATIC_URL to the start of the path. With staticfiles, it calls the url(path) method of the Storage class. In production, with ManifestStaticFilesStorage, it uses staticfiles.json to return a URLs with hashes in the name. For example, here is the HTML that includes the Tumbeast in the 404 page: <div id="beastainer">    </div></p><p>2.17.11 Template tag statici18n</p><p>The tag statici18n is provided by django-statici18n. It works like the static tag, outputing the URL of the localization JavaScript. This is included in <body> of all page via the base template, near the bottom: <script src="{{ statici18n(request.LANGUAGE_CODE) }}"></script></p><p>2.17. Building, collecting, and serving assets 65 Kuma Documentation, Release latest</p><p>2.17.12 Template tags javascript and stylesheet</p><p> django-pipeline provides two template tags, {% javascript(’bundle’) %} and {% stylesheet(’bundle’) %}, that can inject the <script> and <link> elements into a template. Bundling is controlled by the setting PIPELINE[’PIPELINE_ENABLED’] (False for development, True for production). When bundled, the assets are assumed to be processed and collected, so a single element representing the ﬁnal asset URL is inserted. When bundling is off, the assets are assumed to still be in the source form, and multiple HTML elements are inserted into the document. These tags look more like Jinja2 calls then HTML, like these tags from the revision dashboard: {% block js %} {% javascript 'jquery-ui' %} {% javascript 'dashboard' %} {% endblock %}</p><p> django-pipeline supports other output formats. For example, the editor-content bundle is processed with the javascript-array template, which converts the URLs to a format that can be injected into a JavaScript array, such as the conﬁguration script: win.mdn.assets = { css: { 'editor-content': [ {%- stylesheet 'editor-content' %} {%- stylesheet 'editor-locale-%s' % LANG %} ], 'wiki-compat-tables': [{% stylesheet 'wiki-compat-tables' %}] }, js: { 'syntax-prism': [{% javascript 'syntax-prism' %}], 'wiki-compat-tables': [{% javascript 'wiki-compat-tables' %}] } };</p><p>2.17.13 Serving assets with WhiteNoise</p><p>WhiteNoise is a static ﬁle serving application, and is an alternative to serving static assets with nginx, Apache, or from Amazon S3. On Kuma, it is used to serve static assets in development as well as production. It made it easy to serve HTML and related assets on the same HTTP/2 connection. Kuma uses WhiteNoise as a middleware, included as kuma.core.middleware.RestrictedWhiteNoiseMiddleware. This is a wrapper around whitenoise.middleware.WhiteNoiseMiddleware which skips static ﬁle serving if Kuma is acting as the attachments / samples host. In development (DEBUG = True) and testing, WhiteNoise is in “autorefresh” mode, and uses the staticﬁles-ﬁnder. Each web request to /static scans for the ﬁle to use, which can be slow, but will catch any changes made to the ﬁles. In production (DEBUG = False), the ﬁles in STATIC_ROOT (/static) are indexed when the web server starts up. It also determine headers, such as caching headers and the CORS header, that will be sent with the ﬁle. This makes it very fast to serve static ﬁles, but changes after the web server starts will not be noticed. WhiteNoise provides its own Storage classes, that can compress and cache static asset ﬁles. These are currently unused by Kuma, which uses classes based on those provided by django-pipeline.</p><p>66 Chapter 2. Getting started Kuma Documentation, Release latest</p><p>2.17.14 Future</p><p>• Ensure ﬁles that are not meant for visitors are not collected, to speed up development, collecting, and preparing production images. • Remove the CachedFileFinder and PipelineFinder. • Remove django-pipeline, using gulp on the server as well before running ./manage.py collectstatic. • Add django-webpack-loader or similar to integrate React assets</p><p>2.17.15 History</p><p>The staticfiles application was probably part of the Kuma project from the beginning in 2011. In the SCL3 datacenter, one of the ﬁrst steps of a production push was collecting the static ﬁles to a directory on a network drive. This was shared between web servers, so that the new assets were immediately avaiable as the new code was deployed. Because of ﬁle hashing, it was possible to keep old versions of assets along with new versions. These ﬁles were served by Apache. In 2013, staticfiles was used to serve assets in the development <a href="/tags/Vagrant_(software)/" rel="tag">Vagrant</a> environment instead of Apache, so that collectstatic was not needed to see changes. However, CSS ﬁles were converted to Stylus that year, which required compilation for development and deployment. In 2015, several changes were made to prepare for the move from SCL3 to AWS. One change was to move assets from the /media folder, which is traditionally used for user uploads, to the /kuma/static folder. Another was adopting django-pipeline to compile assets, and WhiteNoise to serve them in production. In 2017, MDN hosting moved from SCL3 to AWS. Apache was no longer used to serve assets, and WhiteNoise was used in production as well. This dropped the ability to serve old versions of assets, but a CDN with long caching times mitigated issues around deployments. That same year, the CSS sources were converted from Stylus to Sass. In 2019, the development team decided to adopt new tools such as React and Webpack(ADR-004).</p><p>2.17. Building, collecting, and serving assets 67</p>
                </div>
            </article>
        </div>
    </div>
</div>
<script type="text/javascript" async crossorigin="anonymous" src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-8519364510543070"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.1/jquery.min.js" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script>
    var docId = 'da2850eed855fff32d8e83d1adcea65b';
    var endPage = 1;
    var totalPage = 71;
    var pfLoading = false;
    window.addEventListener('scroll', function () {
        if (pfLoading) return;
        var $now = $('.article-imgview .pf').eq(endPage - 1);
        if (document.documentElement.scrollTop + $(window).height() > $now.offset().top) {
            pfLoading = true;
            endPage++;
            if (endPage > totalPage) return;
            var imgEle = new Image();
            var imgsrc = "//data.docslib.org/img/da2850eed855fff32d8e83d1adcea65b-" + endPage + (endPage > 3 ? ".jpg" : ".webp");
            imgEle.src = imgsrc;
            var $imgLoad = $('<div class="pf" id="pf' + endPage + '"><img src="/loading.gif"></div>');
            $('.article-imgview').append($imgLoad);
            imgEle.addEventListener('load', function () {
                $imgLoad.find('img').attr('src', imgsrc);
                pfLoading = false
            });
            if (endPage < 7) {
                adcall('pf' + endPage);
            }
        }
    }, { passive: true });
</script>
<script>
    var sc_project = 11552861;
    var sc_invisible = 1;
    var sc_security = "b956b151";
</script>
<script src="https://www.statcounter.com/counter/counter.js" async></script>
</html>