Mozilla Services Documentation Release
Tarek Ziade
Sep 11, 2017
Contents
1 How To... 3
2 Services 21
3 Server-side development guide 121
4 Client Development 151
5 Miscellaneous 185
i ii Mozilla Services Documentation, Release
Welcome to the Mozilla Services Documentation front page. This site contains technical information about the various services and products provided by the Mozilla Services Team. The documentation on this site is meant to be more declarative and static in nature. Other information surrounding the products and services detailed on this site can be found at: • Mozilla Services Team Wiki • Mozilla Developer Network (MDN) The wiki contains team info and updates. MDN contains content specific for helping developers. To contribute to this site, see About this Website.
Contents 1 Mozilla Services Documentation, Release
2 Contents CHAPTER 1
How To...
This section contains simple how-tos. If you want to see a new one, let us know in the mailing list.
Run your own Sync-1.1 Server
The Firefox Sync Server is deployed on our systems using RPM packaging, and we don’t provide any other packaging or publish official RPMs yet. The easiest way to install a Sync Server is to checkout our repository and run a build in-place. Once this is done, Sync can be run behind any Web Server that supports the WSGI protocol.
Note: These instructions are for the sync server protocol used by Firefox 28 and earlier. Firefox 29 and later include a new sync service that is incompatible with this server. For a server compatible with Firefox 29 and later, see Run your own Sync-1.5 Server.
Prerequisites
The various parts are using Python 2.6 and Virtualenv. Make sure your system has them. Or install them: • Python 2.6 downloads: http://python.org/download/releases/2.6.6 • Virtualenv: http://pypi.python.org/pypi/virtualenv To run the server, you will also need to have these packages installed: • python-dev • make • mercurial • sqlite3 • openssl-dev
3 Mozilla Services Documentation, Release
For example, under a fresh Ubuntu, you can run this command to meet all requirements:
$ sudo apt-get install python-dev mercurial sqlite3 python-virtualenv libssl-dev
Building the server
Get the latest version at https://hg.mozilla.org/services/server-full and run the build command:
$ hg clone https://hg.mozilla.org/services/server-full $ cd server-full $ make build
This command will create an isolated Python environment and pull all the required dependencies in it. A bin directory is created and contains a paster command that can be used to run the server, using the built-in web server.
Note: Occasionally the build may fail due to network issues that make PyPI inaccessible. If you receive an error about “Could not find suitable distribution”, try waiting a little while and then running the build again.
If you like, you can run the testsuite to make sure everything is working properly:
$ make test
If this gives you an error about “pysqlite2”, you may need to install the “pysqlite” package like so:
$ ./bin/pip install pysqlite
Basic Configuration
The server is configured using an ini-like file to specify various runtime settings. The file “etc/sync.conf” will provide a useful starting point”. There is one setting that you must specify before running the server: the client-visible URL for the storage service node. To ensure that the Registration and Node-Assignment flow works correctly, this should be set to the URL at which you will be running the server. Open “etc/sync.conf”, locate and uncomment the following lines:
[nodes] fallback_node= http://localhost:5000/
By default the server is configured to use a SQLite database for the storage and the user APIs, with the database file stored at “/tmp/test.db”. You will almost certainly want to change this to a more permanent location:
[storage] sqluri= sqlite:////path/to/database/file.db
[auth] sqluri= sqlite:////path/to/database/file.db
Alternatively, consider using a different database backend as described in Using MYSQL or LDAP or ....
4 Chapter 1. How To... Mozilla Services Documentation, Release
Running the Server
Now you can run the server using paster and the provided “development.ini” file:
$ bin/paster serve development.ini Starting server in PID 29951. serving on 0.0.0.0:5000 view at http://127.0.0.1:5000
Once the server is launched, you can run the Firefox Sync Wizard and choose http://localhost:5000 as your Firefox Custom Sync Server. You should then see a lot of output in the stdout, which are the calls made by the browser for the initial sync.
Note: As of October 2015, the Mozilla-hosted sync-1.1 service has been decommissioned, so you must perform some additional configuration to let Firefox use a self-hosted key-exchange server; see below.
Configuring Firefox
While the sync setup dialog allows you to specify a custom storage server, it does not provide any UI for specifying a custom key-exchange server. In order to pair with a second device, you will need to go to “about:config”, search for “jpake.serverURL” and change its value to the URL of your server with a “/jpake” suffix: • services.sync.jpake.serverURL: http://localhost:5000/jpake
Updating the server
You should periodically update your code to make sure you’ve got the latest fixes. The following commands will update server-full in place:
$ cd /path/to/server-full $ hg pull $ hg update $ make build
By default, the build command will checkout the latest released tags for each server product. If you need access to a fix that has not yet been released (or if you just want to live on the bleeding edge) then you can build the development channel like so:
$ make build CHANNEL=dev
Note: Due to a change in how authentication is handled, users upgrading from a build made prior to January 2012 may need to migrate user accounts into a new database table. To do so: 1. Check that the [auth] section in your config file is using the “services.user.sql.SQLUser” backend. 2. Check if your database contains a “users” table. 3. If so, use the following migration script to move data into the “user” table:
deps/server-core/migrations/auth.sql_to_user.sql_migration.txt
1.1. Run your own Sync-1.1 Server 5 Mozilla Services Documentation, Release
Security Notes
File Permissions
The default configuration of the server uses a file-based sqlite database, so you should carefully check that the permis- sions on this file are appropriate for your setup. The file and its containing directory should be writable by the user under which the server is running, and inaccessible to other users on the system. You may like to set the umask of the server process to ensure that any files it creates are readable only by the appropriate user. For example:
$ umask 007 $ bin/paster serve development.ini
Disabling New Users
The default configuration of the server allows new users to create an account through Firefox’s builtin setup screen. This is useful during initial setup, but it means that anybody could sync against your server if they know its URL. You can disable creation of new accounts by setting auth.allow_new_users to false in the config file:
[auth] allow_new_users= false
Using MYSQL or LDAP or ...
Instead of SQLite, you can use alternative backends: • Open-LDAP to store the users • A SQLAlchemy-compatible database, to store the sync data and/or the users Sync has been tested on MySQL and Postgres. In order to use a specific Database, you need to install the required headers, and the required Python library in the local Python environment. See http://www.sqlalchemy.org/docs/core/engines.html#supported-dbapis For example, to run everything in MySQL: 1. install libmysqlclient-dev and mysql-server 2. install Mysql-Python by running bin/easy_install Mysql-Python 3. change the configuration file located at etc/sync.conf For #3, see Configuring the application. For SQL databases, the code will create three tables: • user: contains the user accounts, mapping email to numeric id. • collections: contains collection names for each user, by numeric id. • wbo: contains individual sync records for each user, by numeric id.
6 Chapter 1. How To... Mozilla Services Documentation, Release
Running behind a Web Server
The built-in server should not be used in production, as it does not really support a lot of load. If you want to set up a production server, you can use different web servers that are compatible with the WSGI protocol. For example: • Apache combined with mod_wsgi • NGinx with Gunicorn or uWSGI • lighttpd with flup, using the fcgi or scgi protocol
Note: Remember, you must set the nodes.fallback_node option to the client-visible URL of your sync server. For example, if your server will be located at http://example.com/ff-sync/, the fallback node should be set to this value in your config file: [nodes] fallback_node= http://example.com/ff-sync/
Apache + mod_wsgi
Here’s an example of an Apache 2.2 setup that uses mod_wsgi:
Here’s the equivalent setup for Apache 2.4, which uses a different syntax for acess control:
1.1. Run your own Sync-1.1 Server 7 Mozilla Services Documentation, Release
We provide a sync.wsgi file for your convenience in the repository. Before running Apache, edit the file and check that it loads the the right .ini file with its full path.
Nginx + Gunicorn
Tested with debian stable/squeeze 1. First install gunicorn in the server-full python version:
$ cd /usr/src/server-full $ bin/easy_install gunicorn
2. Then enable gunicorn in the developement.ini:
[server:main] use= egg:gunicorn host= 127.0.0.1 port= 5000 workers=2 timeout= 60
3. Edit etc/sync.conf:
[nodes] fallback_node= https://www.yourserver.net/some/path/
4. Finally edit your nginx vhost file:
server { listen 443 ssl; server_name sync.example.com;
ssl_certificate /path/to/your.crt; ssl_certificate_key /path/to/your.key;
location / { proxy_pass_header Server; proxy_set_header Host $http_host; proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; proxy_connect_timeout 10; proxy_read_timeout 120; proxy_pass http://localhost:5000/; } }
5. After restarting your nginx and server-full you should be able to use the sync server behind your nginx installa- tion lighttpd + flup + fcgi
Tested under Gentoo. 1. Make sure you have the following packages installed: • virtualenv
8 Chapter 1. How To... Mozilla Services Documentation, Release
• mercurial With Gentoo use:
emerge-avuDN virtualenv mercurial
1. Install flup in the server-full python version:
$ cd /usr/src/server-full $ bin/easy_install flup
4. I had to edit the Makefile to take out the memcache dependency. YMMV. 5. Edit development.ini:
[server:main] use= egg:Flup #fcgi_thread host= 0.0.0.0 port= 5000
Be sure to remove the “use_threadpool” and “threadpool_workers” options from this section, since fcgi does not support them. 6. Edit etc/sync.conf:
[storage] backend= syncstorage.storage.sql.SQLStorage sqluri= sqlite:////usr/src/server-full/weave_storage create_tables= true
[auth] backend= services.user.sql.SQLUser sqluri= sqlite:////usr/src/server-full/weave_user create_tables= true
[nodes] fallback_node= https://www.yourserver.net/some/path/
7. Edit your lighttpd.conf:
server.modules+=("mod_fastcgi") fastcgi.server=("/some/path"=>(( "host"=>"127.0.0.1", "port"=> 5000, "idle-imeout"=> 32, "check-local"=>"disable", "disable-time"=>1, "fix-root-scriptname"=>"enable" )) )
Be sure to not add a trailing slash after “/some/path”, otherwise you will get a 404 error. 8. Start the Python server:
/usr/src/server-full/paster serve/usr/src/server-full/development.ini--daemon
9. Restart your lighttpd:
1.1. Run your own Sync-1.1 Server 9 Mozilla Services Documentation, Release
/etc/init.d/lighttpd restart
Troubleshooting
Most issues with the server are caused by bad configuration. If your server does not work properly, the first thing to do is to visit about:sync-log in Firefox to see if there’s any error. You will see a lot of logs and if the sync failed probably an error.
Misconfigured storage node
If the last successful call is finishing like this:
2011-02-24 11:17:57 Net.Resource DEBUG GET success 200 http://server/user/1.
˓→0/.../node/weave
But is not followed by:
2011-02-24 11:17:57 Service.Main DEBUG cluster value= http://server/ 2011-02-24 11:17:57 Service.Main DEBUG Caching URLs under storage user base:
˓→http://server/.../ 2011-02-24 11:17:57 Net.Resource DEBUG GET success 200 http://server/.../
˓→info/collections
It probably means that your server fallback_node option is not properly configured. See the previous section.
Getting a lot of 404
Check your server logs and make sure your VirtualHost is properly configured. Looking at the server log might help.
Getting some 500 errors
Check your server logs and look for some tracebacks. Also, make sure your server-full code is up-to-date by running make build Some common errors: • KeyError: “Unknown fully qualified name for the backend: ‘sql”’ This error means that your backend configuration is outdated. Use the fully qualified names described in the previous sections. • Various datatype-related errors This could indicate that your webserver’s own authentication system is interacting badly with the sync server’s own system. You may need to e.g. disable apache’s basic auth system.
Firefox says the server URL is invalid
Check that you have entered the full URL, including a leading “http://” or “https://” component. Check that you’re not running your server on a port number that is commonly used for other services, such as port 22 (used by ssh) or port 6000 (used by X11). Firefox may prevent outgoing HTTP connections to these ports for security reasons.
10 Chapter 1. How To... Mozilla Services Documentation, Release
The current list of blocked ports can be viewed at http://dxr.mozilla.org/mozilla-central/netwerk/base/src/nsIOService. cpp.html#l70.
Can’t get it to work
Ask for help: • on IRC (irc.mozilla.org) in the #sync channel • in our Mailing List: https://mail.mozilla.org/listinfo/sync-dev
Run your own Sync-1.5 Server
The Firefox Sync Server is deployed on our systems using RPM packaging, and we don’t provide any other packaging or publish official RPMs yet. The easiest way to install a Sync Server is to checkout our repository and run a build in-place. Once this is done, Sync can be run behind any Web Server that supports the WSGI protocol.
Important Notes
These instructions are for the sync-1.5 server protocol used by the the new sync service in Firefox 29 and later. For a server compatible with earlier versions of Firefox, see Run your own Sync-1.1 Server. The new sync service uses Firefox Accounts for user authentication, which is a separate service and is not covered by this guide.
Note: By default, a server set up using this guide will defer authentication to the Mozilla-hosted accounts server at https://accounts.firefox.com.
You can safely use the Mozilla-hosted Firefox Accounts server in combination with a self-hosted sync storage server. The authentication and encryption protocols are designed so that the account server does not know the user’s plaintext password, and therefore cannot access their stored sync data. Alternatively, you can also Run your own Firefox Accounts Server to control all aspects of the system. The process for doing so is currently very experimental and not well documented.
Prerequisites
The various parts are using Python 2.7 and Virtualenv. Make sure your system has them, or install them: • Python 2.7 downloads: http://python.org/download/releases/2.7.6 • Virtualenv: http://pypi.python.org/pypi/virtualenv To build and run the server, you will also need to have these packages installed: • python-dev • make • git • c and c++ compiler
1.2. Run your own Sync-1.5 Server 11 Mozilla Services Documentation, Release
For example, under a fresh Ubuntu, you can run this command to meet all requirements:
$ sudo apt-get install python-dev git-core python-virtualenv g++
Building the server
Get the latest version at https://github.com/mozilla-services/syncserver and run the build command:
$ git clone https://github.com/mozilla-services/syncserver $ cd syncserver $ make build
This command will create an isolated Python environment and pull all the required dependencies in it. A local/bin directory is created and contains a gunicorn command that can be used to run the server. If you like, you can run the testsuite to make sure everything is working properly:
$ make test
Basic Configuration
The server is configured using an ini-like file to specify various runtime settings. The file “syncserver.ini” will provide a useful starting point. There is one setting that you must specify before running the server: the client-visible URL for the service. Open ”./syncserver.ini” and locate the following lines:
[syncserver] public_url= http://localhost:5000/
The default value of “public_url” will work for testing purposes on your local machine. For final deployment, change it to the external, publicly-visible URL of your server. By default the server will use an in-memory database for storage, meaning that any sync data will be lost on server restart. You will almost certainly want to configure a more permanent database, which can be done with the “sqluri” setting:
[syncserver] sqluri= sqlite:////path/to/database/file.db
This setting will accept any SQLAlchemy database URI; for example the following would connect to a mysql server:
[syncserver] sqluri= pymysql://username:password @db.example.com/sync
Running the Server
Now you can run the server using gunicorn and the provided “syncserver.ini” file. The simplest way is to use the Makefile like this:
$ make serve
Or if you’d like to pass additional arguments to gunicorn, like this:
12 Chapter 1. How To... Mozilla Services Documentation, Release
$ local/bin/gunicorn --threads 4 --paste syncserver.ini
Once the server is launched, you need to tell Firefox about its location. To configure desktop Firefox to talk to your new Sync server, go to “about:config”, search for “iden- tity.sync.tokenserver.uri” and change its value to the URL of your server with a path of “token/1.0/sync/1.5”: • identity.sync.tokenserver.uri: http://sync.example.com/token/1.0/sync/1.5 Alternatively, if you’re running your own Firefox Accounts server, and running Firefox 52 or later, see the documen- tation on how to Run your own Firefox Accounts Server for how to configure your client for both Sync and Firefox Accounts with a single preference. Since Firefox 33, Firefox for Android has supported custom sync servers. To configure Android Firefox 44 and later to talk to your new Sync server, just set the “identity.sync.tokenserver.uri” exactly as above before signing in to Firefox Accounts and Sync on your Android device. Important: after creating the Android account, changes to “identity.sync.tokenserver.uri” will be ignored. (If you need to change the URI, delete the Android account using the Settings > Sync > Disconnect... menu item, update the pref, and sign in again.) Non-default TokenServer URLs are displayed in the Settings > Sync panel in Firefox for Android, so you should be able to verify your URL there. Prior to Firefox 44, a custom add-on was needed to configure Firefox for Android. For Firefox 43 and earlier, see the blog post How to connect Firefox for Android to self-hosted Firefox Account and Firefox Sync servers. (Prior to Firefox 42, the TokenServer preference name for Firefox Desktop was “services.sync.tokenServerURI”. While the old preference name will work in Firefox 42 and later, the new preference is recommended as the old preference name will be reset when the user signs out from Sync causing potential confusion.)
Further Configuration
Once the server is running and Firefox is syncing successfully, there are further configuration options you can tweak in the “syncserver.ini” file. The “secret” setting is used by the server to generate cryptographically-signed authentication tokens. It is blank by default, which means the server will randomly generate a new secret at startup. For long-lived server installations this should be set to a persistent value, generated from a good source of randomness. An easy way to generate such a value on posix-style systems is to do: $ head -c 20 /dev/urandom | sha1sum db8a203aed5fe3e4594d4b75990acb76242efd35 - Then copy-paste the value into the config file like so:
[syncserver] ...other settings... secret= db8a203aed5fe3e4594d4b75990acb76242efd35
The “allow_new_users” setting controls whether the server will accept requests from previously-unseen users. It is allowed by default, but once you have configured Firefox and successfully synced with your user account, additional users can be disabled by setting:
[syncserver] ...other settings... allow_new_users= false
1.2. Run your own Sync-1.5 Server 13 Mozilla Services Documentation, Release
Updating the server
You should periodically update your code to make sure you’ve got the latest fixes. The following commands will update syncserver in place:
$ cd /path/to/syncserver $ git stash # to save any local changes to the config file $ git pull # to fetch latest updates from github $ git stash pop # to re-apply any local changes to the config file $ make build # to pull in any updated dependencies
Running behind a Web Server
The built-in server should not be used in production, as it does not really support a lot of load. If you want to set up a production server, you can use different web servers that are compatible with the WSGI protocol. For example: • Apache combined with mod_wsgi • NGinx with Gunicorn or uWSGI
Note: Remember, you must set the syncserver.public_url option to the client-visible URL of your server. For example, if your server will be located at http://example.com/ff-sync/, the public_url should be set to this value in your config file:
[syncserver] public_url= http://example.com/ff-sync/
Apache + mod_wsgi
Here’s an example of an Apache 2.2 setup that uses mod_wsgi:
˓→path/to/syncserver/local/lib/python2.7/site-packages/ WSGIPassAuthorization On WSGIScriptAlias//path/to/syncserver/syncserver.wsgi CustomLog/var/log/apache2/example.com-access.log combined ErrorLog/var/log/apache2/example.com-error.log
Here’s the equivalent setup for Apache 2.4, which uses a different syntax for access control:
14 Chapter 1. How To... Mozilla Services Documentation, Release
˓→path/to/syncserver/local/lib/python2.7/site-packages/ WSGIPassAuthorization On WSGIScriptAlias//path/to/syncserver/syncserver.wsgi CustomLog/var/log/apache2/example.com-access.log combined ErrorLog/var/log/apache2/example.com-error.log
We provide a syncserver.wsgi file for your convenience in the repository. Before running Apache, edit the file and check that it loads the the right .ini file with its full path.
Nginx + Gunicorn
Tested with debian stable/squeeze 1. First install gunicorn in the syncserver python environment:
$ cd /usr/src/syncserver $ local/bin/easy_install gunicorn
2. Then enable gunicorn in the syncserver.ini file:
[server:main] use= egg:gunicorn host= 127.0.0.1 port= 5000 workers=2 timeout= 60
3. Finally edit your nginx vhost file:
server { listen 443 ssl; server_name sync.example.com;
ssl_certificate /path/to/your.crt; ssl_certificate_key /path/to/your.key;
location / { proxy_set_header Host $http_host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_redirect off; proxy_read_timeout 120; proxy_connect_timeout 10; proxy_pass http://127.0.0.1:5000/; } }
1.2. Run your own Sync-1.5 Server 15 Mozilla Services Documentation, Release
5. After restarting your nginx and syncserver you should be able to use the sync server behind your nginx installa- tion
Note: If you see errors about a mismatch between public_url and application_url, you may need to tell gunicorn that it should trust the X-Forwarded-Proto header being sent by nginx. Add the following to the gunicorn configuration in syncserver.ini: forwarded_allow_ips= *
Note: If you see errors about “client sent too long header line” in your nginx logs, you may need to configure nginx to allow large client header buffers by adding this to the nginx config: large_client_header_buffers48k;
Nginx + uWSGI
1. Install uWSGI and its Python 2 plugin 2. Start it with the following options:
uwsgi--plugins python27--manage-script-name \ --mount/
3. Use the following nginx configuration:
location/
Things that still need to be Documented
• how to restrict new-user signups • how to interoperate with a self-hosted accounts server • periodic pruning of expired sync data
Asking for help
Don’t hesitate to jump online and ask us for help: • on IRC (irc.mozilla.org) in the #sync channel • in our Mailing List: https://mail.mozilla.org/listinfo/sync-dev
16 Chapter 1. How To... Mozilla Services Documentation, Release
Run your own Firefox Accounts Server
The Firefox Accounts server is deployed on our systems using RPM packaging, and we don’t provide any other packaging or publish official builds yet.
Note: This guide is preliminary and vastly incomplete. If you have any questions or find any bugs, please don’t hesitate to drop by the IRC channel or mailing list and let us know.
Note: You might also be interested in this Docker-based self-hosting guide (use at your own risk).
The Firefox Accounts server is hosted in git and requires nodejs. Make sure your system has these, or install them: • git: http://git-scm.com/downloads • nodejs: http://nodejs.org/download A self-hosted Firefox Accounts server requires two components: an auth-server that manages the accounts database, and a content-server that hosts a web-based user interface. Clone the fxa-auth-server repository and follow the README to deploy your own auth-server: • https://github.com/mozilla/fxa-auth-server/ Clone the fxa-content-server repository and follow the README to deploy your own content-server: • https://github.com/mozilla/fxa-content-server/ Now direct Firefox to use your servers rather than the default, Mozilla-hosted ones. The procedure varies a little between desktop and mobile Firefox, and may not work on older versions of the browser. For desktop Firefox version 52 or later: • Enter “about:config” in the URL bar • Right click anywhere on the “about:config” page and choose New > String • Enter “identity.fxaccounts.autoconfig.uri” for the name, and your content-server URL for the value. • Restart Firefox for the change to take effect. Note that this must be set prior to loading the sign-up or sign-in page in order to take effect, and its effects are reset on sign-out. For Firefox for iOS version 9.0 or later: • Go to Settings. • Tap on the Version number 5 times. • Tap on “Advance Account Settings” • Enter your content-server URL • Toggle “Use Custom Account Service” to on. For Firefox for Android version 44 or later: • Enter “about:config” in the URL bar, • Search for items containing “fxaccounts”, and edit them to use your self-hosted URLs: – use your auth-server URL to replace “api.accounts.firefox.com” in the following settings: - iden- tity.fxaccounts.auth.uri
1.3. Run your own Firefox Accounts Server 17 Mozilla Services Documentation, Release
– use your content-server URL to replace “accounts.firefox.com” in the following settings: - iden- tity.fxaccounts.remote.webchannel.uri - webchannel.allowObject.urlWhitelist – optionally, use your oauth- and profile-server URLs to replace “{oauth,profile}.accounts.firefox.com” in - identity.fxaccounts.remote.profile.uri - identity.fxaccounts.remote.oauth.uri Important: after creating the Android account, changes to “identity.fxaccounts” prefs will be ignored. (If you need to change the prefs, delete the Android account using the Settings > Sync > Disconnect... menu item, update the pref(s), and sign in again.) Non-default Firefox Account URLs are displayed in the Settings > Sync panel in Firefox for Android, so you should be able to verify your URL there. Since the Mozilla-hosted sync servers will not trust assertions issued by third-party accounts servers, you will also need to run your own sync-1.5 server. Please note that the fxa-content-server repository includes graphics and other assets that make use of Mozilla trade- marks. If you are doing anything other than running unmodified copies of the software for personal use, please review the Mozilla Trademark Policy and Mozilla Branding Guidelines: • https://www.mozilla.org/en-US/foundation/trademarks/policy/ • http://www.mozilla.org/en-US/styleguide/identity/mozilla/branding/ You can ask for help: • on IRC (irc.mozilla.org) in the #fxa channel • in our Mailing List: https://mail.mozilla.org/listinfo/dev-fxacct
Configure your Sync server for TLS
Firefox for Android versions 39 and up request the following protocols and cipher suites, depending on the Android OS version. The use of AES128 in preference to AES256 is driven by power and CPU concerns.
Cipher Suites
Android 20+:
• TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA • TLS_DHE_RSA_WITH_AES_128_CBC_SHA • TLS_RSA_WITH_AES_128_CBC_SHA
18 Chapter 1. How To... Mozilla Services Documentation, Release
Android 11+:
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA • TLS_DHE_RSA_WITH_AES_128_CBC_SHA • TLS_RSA_WITH_AES_128_CBC_SHA
Android 9+ (Gingerbread):
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA • TLS_DHE_DSS_WITH_AES_128_CBC_SHA • TLS_DHE_RSA_WITH_AES_128_CBC_SHA • TLS_RSA_WITH_AES_128_CBC_SHA
Protocols
API 9 through 15 support only TLSv1.0. Modern versions of Android support all versions of TLS, v1.0 through v1.2. We intend to eliminate TLSv1.0 on suitable Android versions as soon as possible. No version of Firefox for Android beyond 38 supports SSLv3 for Sync.
1.4. Configure your Sync server for TLS 19 Mozilla Services Documentation, Release
20 Chapter 1. How To... CHAPTER 2
Services
This section contains documentation about the services we operate, like their API specifications If you want to see the big picture, check Overview of the services. Or jump to the detailed documentation for each service:
Firefox Accounts Server
The Firefox Accounts server provides a centralized database of all user accounts for accessing Mozilla-hosted services. It replaces the sync-specific registration and secure-registration services. Firefox Accounts support is included in Firefox version 29 and later. By default, Firefox will use Mozilla’s hosted accounts server at https://accounts.firefox.com. This configuration will work well for most use-cases, including for those who want to self-host a storage server. User who want to minimize their dependency on Mozilla-hosted services may also self-host an accounts server, but this setup is incompatible with other Mozilla-hosted services. • Protocol documentation: https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md • API server code: https://github.com/mozilla/fxa-auth-server/ • Web interface code: https://github.com/mozilla/fxa-content-server/
Storage Service
Goal of the Service
The storage server provides web services that can be used to store and retrieve any kind of data. It’s used to power Firefox Sync. If you are using our hosted services with your own client code, please read the Term of Services.
21 Mozilla Services Documentation, Release
Documentation content
Storage API v1.0
This document describes the Sync Server Storage API, version 1.0. It has been replaced by Storage API v1.1.
Weave Basic Object (WBO)
A Weave Basic Object is the generic wrapper around all items passed into and out of the Weave server. The Weave Basic Object has the following fields: Parameter Default Max Description id required 64 An identifying string. For a user, the id must be unique for a WBO within a collection, though ob- jects in different collec- tions may have the same ID. Ids should be ASCII and not contain commas. parentid none 64 The id of a parent ob- ject in the same collection. This allows for the cre- ation of hierarchical struc- tures (such as folders). predecessorid none 64 The id of a predecessor in the same collection. This allows for the cre- ation of linked-list-esque structures. modified time submitted float The last-modified date, in 2 decimal places seconds since 1970-01-01 (UNIX epoch time see1). Set by the server. sortindex none 256K A string containing a JSON structure encap- sulating the data of the record. This structure is defined separately for each WBO type. Parts of the structure may be encrypted, in which case the structure should also specify a record for decryption. payload none 256K Weave Basic Objects and all data passed into the Weave Server should be utf-8 encoded. Sample:
{ "id":"B1549145-55CB-4A6B-9526-70D370821BB5",
1 http://www.ecma-international.org/publications/standards/Ecma-262.htm ecma-262
22 Chapter 2. Services Mozilla Services Documentation, Release
"parentid":"88C3865F-05A6-4E5C-8867-0FAC9AE264FC", "modified":"2454725.98", "payload":"{ \"encryption\":\"http://server/prefix/version/user/crypto-meta/
˓→B1549145-55CB-4A6B-9526-70D370821BB5\", \"data\": \"a89sdmawo58aqlva.
˓→8vj2w9fmq2af8vamva98fgqamff...\"}" }
Collections
Each WBO is assigned to a collection with other related WBOs. Collection names may only contain alphanumeric characters, period, underscore and hyphen. Collections supported at this time are: • bookmarks • history • forms • prefs • tabs • passwords Additionally, the following collections are supported for internal Weave client use: • clients • crypto • keys • meta
URL Semantics
Weave URLs follow, for the most part, REST semantics. Request and response bodies are all JSON-encoded. The URL for Weave Storage requests is structured as follows: https://
Component Mozilla Default Description server name (defined by user account the hostname of the server node) pathname (none) the prefix associated with the service on the box version 1.0 The API version. May be integer or decimal username (none) The name of the object (user) to be manipulated further (none) The additional function information as defined in the paths instruction below Weave uses HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not match the username in the path, the server will issue an Error Response. The Weave API has a set of Weave Response Codes to cover errors in the request or on the server side. The format of a successful response is defined in the appropriate request method section.
2.2. Storage Service 23 Mozilla Services Documentation, Release
GET https://*server*/*pathname*/*version*/*username*/info/collections Returns a hash of collections associated with the account, along with the last modified timestamp for each collection. https://*server*/*pathname*/*version*/*username*/info/collection_counts Returns a hash of collections associated with the account, along with the total number of items for each collection. https://*server*/*pathname*/*version*/*username*/info/quota Returns a tuple containing the user’s current usage (in K) and quota. https://*server*/*pathname*/*version*/*username*/storage/*collection* Returns a list of the WBO ids contained in a collection. This request has additional optional parameters: • ids: Returns the ids for objects in the collection that are in the provided comma-separated list. • predecessorid: Returns the ids for objects in the collection that are directly preceded by the id given. Usually only returns one result. • parentid: Returns the ids for objects in the collection that are the children of the parent id given. • older: Returns only ids for objects in the collection that have been last modified before the date given. • newer: Returns only ids for objects in the collection that have been last modified since the date given. • full: If defined, returns the full WBO, rather than just the id. • index_above: If defined, only returns items with a higher sortindex than the value specified. • index_below: If defined, only returns items with a lower sortindex than the value specified. • limit: Sets the maximum number of ids that will be returned. • offset: Skips the first n ids. For use with the limit parameter (required) to paginate through a result set. • sort:: sorts before getting - ‘oldest’ - Orders by modification date (oldest first) - ‘newest’ - Orders by modification date (newest first) - ‘index’ - Orders by the sortindex descending (highest weight first) https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id* Returns the WBO in the collection corresponding to the requested id
Alternate Output Formats
Two alternate output formats are available for multiple record GET requests. They are triggered by the presence of the appropriate format in the Accept header (with application/whoisi taking precedence) • application/whoisi: each record consists of a 32-bit integer, defining the length of the record, followed by the json record for a wbo • application/newlines: each record is a separate json object on its own line. Newlines in the body of the json object are replaced by ‘u000a’
24 Chapter 2. Services Mozilla Services Documentation, Release
APIs
PUT
https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id* Adds the WBO defined in the request body to the collection. If the WBO does not contain a payload, it will only update the provided metadata fields on an already defined object. The server will return the timestamp associated with the modification.
POST
https://*server*/*pathname*/*version*/*username*/storage/*collection* Takes an array of WBOs in the request body and iterates over them, effectively doing a series of atomic PUTs with the same timestamp. Returns a hash of successful and unsuccessful saves, including guidance as to possible errors: {“modified”:1233702554.25,”success”:[”{GXS58IDC}12”,”{GXS58IDC}13”,”{GXS58IDC}15”,”{GXS58IDC}16”,”{GXS58IDC}18”,”{GXS58IDC}19”],”failed”:{“{GXS58IDC}11”:[”invalid parentid”],”{GXS58IDC}14”:[”invalid parentid”],”{GXS58IDC}17”:[”invalid paren- tid”],”{GXS58IDC}20”:[”invalid parentid”]}}
DELETE
https://*server*/*pathname*/*version*/*username*/storage/*collection* Deletes the collection and all contents. Additional request parameters may modify the selection of which items to delete: • ids: Deletes the ids for objects in the collection that are in the provided comma-separated list. • parentid: Only deletes objects in the collection that are the children of the parent id given. • older: Only deletes objects in the collection that have been last modified before the date given. • newer: Only deletes objects in the collection that have been last modified since the date given. • limit: Sets the maximum number of objects that will be deleted. • offset: Skips the first n objects in the defined set. Must be used with the limit parameter. [This function is not currently operational in the mysql implementation] • sort: Sorts items before deletion - ‘oldest’ - Orders by modification date (oldest first) - ‘newest’ - Orders by modification date (newest first) - ‘index’ - Orders by the sortindex (ordered lists) https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id* Deletes the WBO at the location given All delete requests return the timestamp of the action. https://*server*/*pathname*/*version*/*username*/storage* Deletes all records for the user. Will return a precondition error unless an X-Confirm-Delete header is included. All delete requests return the timestamp of the action.
2.2. Storage Service 25 Mozilla Services Documentation, Release
General Weave Headers
X-Weave-Backoff Indicates that the server is under heavy load or has suffered a failure and the client should not try again for the specified number of seconds (usually 1800) X-If-Unmodified-Since On any write transaction (PUT, POST, DELETE), this header may be added to the request, set to a times- tamp. If the collection to be acted on has been modified since the timestamp given, the request will fail. X-Weave-Alert This header may be sent back from any transaction, and contains potential warning messages, information, or other alerts. The contents are intended to be human-readable. X-Weave-Timestamp This header will be sent back with all requests, indicating the current timestamp on the server. If the request was a PUT or POST, this will also be the modification date of any WBOs submitted or modified. X-Weave-Records If supported by the db, this header will return the number of records total in the request body of any multiple-record GET request.
Storage API v1.1
The Storage server provides web services that can be used to store and retrieve Weave Basic Objects or WBOs organized into collections.
Weave Basic Object
A Weave Basic Object (WBO) is the generic JSON wrapper around all items passed into and out of the storage server. Like all JSON, Weave Basic Objects need to be UTF-8 encoded. WBOs have the following fields:
26 Chapter 2. Services Mozilla Services Documentation, Release
Parameter Default Type/Max Description id required string 64 An identifying string. For a user, the id must be unique for a WBO within a collection, though ob- jects in different collec- tions may have the same ID. This should be exactly 12 characters from the base64url alphabet. While this isn’t enforced by the server, the Firefox client expects this in most cases. modified time submitted float 2 decimal places The last-modified date, in seconds since 01/01/197023 sortindex none integer An integer indicating the relative importance of this item in the collection. payload none string 256k A string containing a JSON structure encap- sulating the data of the record. This structure is defined separately for each WBO type. Parts of the structure may be encrypted, in which case the structure should also specify a record for decryption. ttl none integer The number of seconds to keep this record. After that time, this item will not be returned. parentid1 none string 64 The id of a parent ob- ject in the same collection. This allows for the cre- ation of hierarchical struc- tures (such as folders). predecessorid1 none string 64 The id of a predecessor in the same collection. This allows for the cre- ation of linked-list-esque structures. Sample:
{ "id":"-F_Szdjg3GzY", "modified":1278109839.96, "sortindex":140,
2 See ecma-262: http://www.ecma-international.org/publications/standards/Ecma-262.htm 3 Set automatically by the server 1 This is deprecated and likely going away in future versions
2.2. Storage Service 27 Mozilla Services Documentation, Release
"payload":"{\"ciphertext\":\"e2zLWJYX\/
˓→iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/
˓→uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/
˓→ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/
˓→AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\
˓→"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}" }
Collections
Each WBO is assigned to a collection with other related WBOs. Collection names may only contain alphanumeric characters, period, underscore and hyphen. Default Mozilla collections are: • bookmarks • history • forms • prefs • tabs • passwords Additionally, the following collections are supported for internal storage client use: • clients • crypto • keys • meta
URL semantics
Storage URLs follow, for the most part, REST semantics. Request and response bodies are all JSON-encoded. The URL for Weave Storage requests is structured as follows:
https://
Component Mozilla Default Description server name defined by user account the hostname of the server pathname (none) the prefix associated with the service on the box version 1.1 The API version. username (none) the name of the object (user) to be manipulated further instruction (none) The additional function information as defined in the paths below Certain functions use HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not match the username in the path, the server will issue an Error Response. The User API has a set of Response codes to cover errors in the request or on the server side. The format of a successful response is defined in the appropriate request method section.
28 Chapter 2. Services Mozilla Services Documentation, Release
APIs
GET https://server/pathname/version/username/info/collections Returns a hash of collections associated with the account, along with the last modified timestamp for each collection. GET https://server/pathname/version/username/info/collection_usage Returns a hash of collections associated with the account, along with the data volume used for each (in KB). GET https://server/pathname/version/username/info/collection_counts Returns a hash of collections associated with the account, along with the total number of items in each collection. GET https://server/pathname/version/username/info/quota Returns a list containing the user’s current usage and quota (in KB). The second value will be null if no quota is defined. GET https://server/pathname/version/username/storage/collection Returns a list of the WBO ids contained in a collection. This request has additional optional parameters: • ids: returns the ids for objects in the collection that are in the provided comma-separated list. • predecessorid: returns the ids for objects in the collection that are directly preceded by the id given. Usually only returns one result.4 • parentid: returns the ids for objects in the collection that are the children of the parent id given.4 • older: returns only ids for objects in the collection that have been last modified before the date given. • newer: returns only ids for objects in the collection that have been last modified since the date given. • full: if defined, returns the full WBO, rather than just the id. • index_above: if defined, only returns items with a higher sortindex than the value specified. • index_below: if defined, only returns items with a lower sortindex than the value specified. • limit: sets the maximum number of ids that will be returned. • offset: skips the first n ids. For use with the limit parameter (required) to paginate through a result set. • sort: sorts the output. • ‘oldest’ - Orders by modification date (oldest first) • ‘newest’ - Orders by modification date (newest first) • ‘index’ - Orders by the sortindex descending (highest weight first) Two alternate output formats are available for multiple record GET requests. They are triggered by the presence of the appropriate format in the Accept header (with application/whoisi taking precedence): • application/whoisi: each record consists of a 32-bit integer, defining the length of the record, fol- lowed by the json record for a WBO • application/newlines: each record is a separate json object on its own line. Newlines in the body of the json object are replaced by ‘u000a’
4 Deprecated in 1.1
2.2. Storage Service 29 Mozilla Services Documentation, Release
GET https://server/pathname/version/username/storage/collection/id Returns the WBO in the collection corresponding to the requested id PUT https://server/pathname/version/username/storage/collection/id Adds the WBO defined in the request body to the collection. If the WBO does not contain a payload, it will only update the provided metadata fields on an already defined object. The server will return the timestamp associated with the modification. POST https://server/pathname/version/username/storage/collection Takes an array of WBOs in the request body and iterates over them, effectively doing a series of atomic PUTs with the same timestamp. Returns a hash of successful and unsuccessful saves, including guidance as to possible errors:
{"modified": 1233702554.25, "success":["{GXS58IDC}12"," {GXS58IDC}13"," {GXS58IDC}15", "{GXS58IDC}16"," {GXS58IDC}18"," {GXS58IDC}19"], "failed":{"{GXS58IDC}11":["invalid parentid"], "{GXS58IDC}14":["invalid parentid"], "{GXS58IDC}17":["invalid parentid"], "{GXS58IDC}20":["invalid parentid"]}}
DELETE https://server/pathname/version/username/storage/collection Deletes the collection and all contents. Additional request parameters may modify the selection of which items to delete: • ids: deletes the ids for objects in the collection that are in the provided comma-separated list. • parentid: only deletes objects in the collection that are the children of the parent id given.4 • older: only deletes objects in the collection that have been last modified before the date given.4 • newer: only deletes objects in the collection that have been last modified since the date given.4 • limit: sets the maximum number of objects that will be deleted.4 • offset: skips the first n objects in the defined set. Must be used with the limit parameter.5 • sort: sorts before deleting4 • ‘oldest’ - Orders by modification date (oldest first) • ‘newest’ - Orders by modification date (newest first) • ‘index’ - Orders by the sortindex (ordered lists) DELETE https://server/pathname/version/username/storage/collection/id Deletes the WBO at the location given DELETE https://server/pathname/version/username/storage Deletes all records for the user. Will return a precondition error unless an X-Confirm-Delete header is included. All delete requests return the timestamp of the action.
5 This function is not currently operational in the mysql implementation
30 Chapter 2. Services Mozilla Services Documentation, Release
Headers
Retry-After When sent together with an HTTP 503 status code, it signifies that the server is undergoing maintenance. The client should not attempt another sync for the number of seconds specified in the header value. X-Weave-Backoff Indicates that the server is under heavy load and the client should not trigger another sync for the number of seconds specified in the header value (usually 1800). X-If-Unmodified-Since On any write transaction (PUT, POST, DELETE), this header may be added to the request, set to a times- tamp in decimal seconds. If the collection to be acted on has been modified since the provided timestamp, the request will fail with an HTTP 412 Precondition Failed status. X-Weave-Alert This header may be sent back from any transaction, and contains potential warning messages, information, or other alerts. The contents are intended to be human-readable. X-Weave-Timestamp This header will be sent back with all requests, indicating the current timestamp on the server. If the request was a PUT or POST, this will also be the modification date of any WBOs submitted or modified. X-Weave-Records If supported by the DB, this header will return the number of records total in the request body of any multiple-record GET request.
HTTP status codes
200 The request was processed successfully. 400 The request itself or the data supplied along with the request is invalid. The response contains a numeric code indicating the reason for why the request was rejected. See Response codes for a list of valid response codes. 401 The username and password are invalid on this node. This may either be caused by a node reassignment or by a password change. The client should check with the auth server whether the user’s node has changed. If it has changed, the current sync is to be aborted and should be retried against the new node. If the node hasn’t changed, the user’s password was changed. 404 The requested resource could not be found. This may be returned for GET and DELETE requests, for non-existent records and empty collections. 503 Indicates, in conjunction with the Retry-After header, that the server is undergoing maintenance. The client should not attempt another sync for the number of seconds specified in the header value. The response body may contain a JSON string describing the server’s status or error.
2.2. Storage Service 31 Mozilla Services Documentation, Release
SyncStorage API v1.5
The SyncStorage API defines a HTTP web service used to store and retrieve simple objects called Basic Storage Objects (BSOs), which are organized into named collections.
Concepts
Basic Storage Object
A Basic Storage Object (BSO) is the generic JSON wrapper around all items passed into and out of the SyncStorage server. Like all JSON documents, BSOs are composed of unicode character data rather than raw bytes and must be encoded for transmission over the network. The SyncStorage service always encodes BSOs in UTF8. Basic Storage Objects have the following fields:
32 Chapter 2. Services Mozilla Services Documentation, Release
Parameter Default Type/Max Description id required string, 64 An identifying string. For a user, the id must be unique for a BSO within a collection, though ob- jects in different collec- tions may have the same ID. BSO ids must only con- tain printable ASCII char- acters. They should be exactly 12 base64-urlsafe characters; while this isn’t enforced by the server, the Firefox client expects it in most cases. modified none float, 2 decimal places The timestamp at which this object was last mod- ified, in seconds since UNIX epoch (1970-01-01 00:00:00 UTC). This is set automatically by the server according to its own clock; any client- supplied value for this field is ignored. sortindex none integer, 9 digits An integer indicating the relative importance of this item in the collection. payload empty string string, at least 256KiB A string containing the data of the record. The structure of this string is defined separately for each BSO type. This spec makes no require- ments for its format. In practice, JSONObjects are common. Servers must support pay- loads up to 256KiB in size. They may ac- cept larger payloads, and advertise their maximum payload size via dynamic configuration. ttl none integer, positive, 9 digits The number of seconds to keep this record. After that time this item will no longer be returned in re- sponse to any request, and it may be pruned from the database. If not specified or null, the record will not expire. This field may be set on write, but is not returned 2.2. Storage Service by the server. 33 Mozilla Services Documentation, Release
Example:
{ "id":"-F_Szdjg3GzX", "modified": 1388635807.41, "sortindex": 140, "payload":"{ \"this is\": \"an example\" }" }
Collections
Each BSO is assigned to a collection with other related BSOs. Collection names may be up to 32 characters long, and must contain only characters from the urlsafe-base64 alphaebet (i.e. alphanumeric characters, underscore and hyphen) and the period. Collections are created implicitly when a BSO is stored in them for the first time. They continue to exist until they are explicitly deleted, even if they no longer contain any BSOs. The default collections used by Firefox to store sync data are: • bookmarks • history • forms • prefs • tabs • passwords The following additional collections are used for internal management purposes by the storage client: • clients • crypto • keys • meta
Timestamps
In order to allow multiple clients to coordinate their changes, the SyncStorage server associates a last-modified time with the data stored for each user. This is a server-assigned decimal value, precise to two decimal places, that is updated from the server’s clock with every modification made to the user’s data. The last-modified time is tracked at three levels of nesting: • The store as a whole has a last-modified time that is updated whenever any change is made to the user’s data. • Each collection has a last-modified time that is updated whenever an item in that collection is modified or deleted. It will always be less than or equal to the overall last-modified time. • Each BSO has a last-modified time that is updated whenever that specific item is modified. It will always be less than or equal to the last-modified time of the containing collection. The last-modified time is guaranteed to be monotonically increasing and can be used for coordination and conflict management as described in Concurrency and Conflict Management.
34 Chapter 2. Services Mozilla Services Documentation, Release
Note that the last-modified time of a collection may be larger than that of any item within in. For example, if all items are deleted from the collection, its last-modified time will be the timestamp of the last deletion.
API Instructions
The SyncStorage data for a given user may be accessed via authenticated HTTP requests to their SyncStorage API endpoint. Request and response bodies are all UTF8-encoded JSON unless otherwise specified. All requests are to URLs of the form:
https://
The user’s SyncStorage endpoint URL can be obtained via the Token Server authentication flow. All requests must be signed using HAWK Authentication credentials obtained from the tokenserver. Error responses generated by the SyncStorage server will, wherever possible, conform to the Response codes defined for the User API. The format of a successful response is defined in the appropriate section of the API Instructions documentation.
General Info
APIs in this section provide a mechanism for high-level interactions with the user’s data store as a whole. GET https://
2.2. Storage Service 35 Mozilla Services Documentation, Release
• max_post_records: the maximum number of records that can be uploaded to a collection in a single POST request. • max_post_bytes: the maximum combined size in bytes of the record payloads that can be uploaded to a collection in a single POST request. • max_total_records: the maximum total number of records that can be uploaded to a collection as part of a batched upload. • max_total_bytes: the maximum total combined size in bytes of the record payloads that can be uploaded to a collection as part of a batched upload. • max_record_payload_bytes: the maximum size of an individual BSO payload, in bytes. DELETE https://
Individual Collection Interaction
APIs in this section provide a mechanism for interacting with a single collection. GET https://
["GXS58IDC_12","GXS58IDC_13","GXS58IDC_15"]
By default only the BSO ids are returned, but full objects can be requested using the full parameter. If the collection does not exist, an empty list is returned. This request has additional optional query parameters: • ids: a comma-separated list of ids. Only objects whose id is in this list will be returned. A maximum of 100 ids may be provided. • newer: a timestamp. Only objects whose last-modified time is strictly greater than this value will be returned. • older: a timestamp. Only objects whose last-modified time is strictly smaller than this value will be returned. • full: any value. If provided then the response will be a list of full BSO objects rather than a list of ids. • limit: a positive integer. At most that many objects will be returned. If more than that many objects matched the query, an X-Weave-Next-Offset header will be returned. • offset: a string, as returned in the X-Weave-Next-Offset header of a previous request using the limit parameter. • sort: sorts the output: – ‘newest’ - orders by last-modified time, largest first – ‘oldest’ - orders by last-modified time, smallest first – ‘index’ - orders by the sortindex, highest weight first
36 Chapter 2. Services Mozilla Services Documentation, Release
The response may include an X-Weave-Records header indicating the total number of records to expect in the body, if the server can efficiently provide this information. If the request included a limit parameter and there were more than that many items matching the query, the response will include an X-Weave-Next-Offset header. This value can be passed back to the server in the offset parameter to efficiently skip over the items that have already been read. See Example: paging through a large set of items for an example. Two output formats are available for multiple-record GET requests. They are triggered by the presence of the appropriate format in the Accept request header and are prioritized in the order listed below: • application/json: the output is a JSON list of the request records, as either string ids or full JSON objects. • application/newlines: the output contains each individual record followed by a newline, as either a string id or a full JSON object. Potential HTTP error responses include: • 400 Bad Request: too many ids were included in the query parameter. GET https://
2.2. Storage Service 37 Mozilla Services Documentation, Release
For backwards-compatibility with existing clients, the server will also treat text/plain input as JSON. Note that the server may impose a limit on the total amount of data included in the request, and/or may decline to process more than a certain number of BSOs in a single request. The default limit on the number of BSOs per request is 100. Successful responses will contain a JSON object with details of success or failure for each BSO. It will have the following keys: • modified: the new last-modified time for the updated items. • success: a (possibly empty) list of ids of BSOs that were successfully stored. • failed: a (possibly empty) object whose keys are the ids of BSOs that were not stored successfully, and whose values are strings describing the reason for the failure. For example:
{ "modified": 1233702554.25, "success":["GXS58IDC_12","GXS58IDC_13","GXS58IDC_15", "GXS58IDC_16","GXS58IDC_18","GXS58IDC_19"], "failed":{"GXS58IDC_11":"invalid ttl"], "GXS58IDC_14":"invalid sortindex"} }
Posted BSOs whose ids do not appear in either “success” or “failed” should be treated as having failed for an unspecified reason. To allow upload of large numbers of items while ensuring that other clients do not sync down inconsistent data, servers may support combining several POST requests into a single “batch” so that all modified BSOs appear to have been submitted at the same time. Batching behaviour is controlled by the following query parameters: • batch: indicates that uploads should be batched together into a single conceptual update. To begin a new batch pass the string ‘true’. To add more items to an existing batch pass a previously-obtained batch identifier. This parameter is ignored by servers that do not support batching. • commit: indicates that the batch should be committed, and all items uploaded to that batch made visible to other clients. If present, it must be the string ‘true’ and the batch query parameter must also be specified. When submitting items for inclusion in a multi-request batch upload, successful responses will have a “202 Accepted” status code, and will contain a JSON object giving the batch identifier rather than modification time, alongside individual success or failure status for each item that was sent. For example:
{ "batch":"OPAQUEBATCHID", "success":["GXS58IDC_12","GXS58IDC_13","GXS58IDC_15", "GXS58IDC_16","GXS58IDC_18","GXS58IDC_19"], "failed":{"GXS58IDC_11":"invalid ttl"], "GXS58IDC_14":"invalid sortindex"} }
The returned value of “batch” can be passed in the “batch” query parameter to add more items to the batch. Items that appear in the “success” list are guaranteed to become available to other clients if and when the batch is successfully committed.
38 Chapter 2. Services Mozilla Services Documentation, Release
Note that the value of “batch” may not be safe to include directly in a URL, and that you need to be sure to encode it first (via JavaScript’s encodeURIComponent, Python’s urllib.quote, or your language’s equivalent). If the server does not support batching, it will ignore the batch parameter and return a “200 OK” response without a batch identifier. The response when committing a batch is identical to that generated by a non-batched request. Note that the semantics of a request with batch=true&commit=true (i.e. starting a batch and immediately committing it) are therefore identical to those of a non-batched request. Note that the server may impose a limit on the total amount of payload data included in a batch, and/or may decline to process more than a certain number of BSOs as part of a single batch. If the uploaded items exceed this limit, the server will produce a 400 Bad Request response with response code 17. Where possible, clients should use the X-Weave-Total-Records* and X-Weave-Total-Bytes headers to signal the expected total size of the uploads, so that oversize batches can be rejected before the items are uploaded. Potential HTTP error responses include: • 400 Bad Request, response code 14: the user has exceeded their storage quota. • 400 Bad Request, response code 17: server size or item-count limit exceeded. • 413 Request Entity Too Large: the request contains more data than the server is willing to process in a single batch. DELETE https://
Request Headers
X-If-Modified-Since
2.2. Storage Service 39 Mozilla Services Documentation, Release
This header may be added to any GET request, set to a decimal timestamp. If the last-modified time of the target resource is less than or equal to the time given in this header, then a 304 Not Modified response will be returned and re-transmission of the unchanged data will be avoided. It is similar to the standard HTTP If-Modified-Since header, but the value is a decimal timestamp rather than a HTTP-format date. If the value of this header is not a valid positive decimal value, or if the X-If-Unmodified-Since header is also present, then a 400 Bad Request response will be returned. X-If-Unmodified-Since This header may be added to any request to a collection or item, set to a decimal timestamp. If the last-modified time of the target resource is greater than the time given, the request will fail with a 412 Precondition Failed response. It is similar to the standard HTTP If-Unmodified-Since header, but the value is a decimal timestamp rather than a HTTP-format date. If the value of this header is not a valid positive decimal value, or if the X-If-Modified-Since header is also present, then a 400 Bad Request response will be returned. X-Weave-Records This header may be sent with multi-record uploads, to indicate the total number of records included in the request. If the server would not accept an upload containing that many records, then a 400 Bad Request response will be returned with response code 17. X-Weave-Bytes This header may be sent with multi-record uploads, to indicate the combined size of payloads in the upload, in bytes. If the server would not accept an upload containing that many bytes, then a 400 Bad Request response will be returned with response code 17. X-Weave-Total-Records This header may be included with a POST request using the batch query parameter, to indicate the total number of records in the batch. If the server would not accept a batch containing that many records, then a 400 Bad Request response will be returned with response code 17. If the value of this header is not a valid positive integer value, or if the request is not operating on a batch, then a 400 Bad Request response will be returned with response code 1. X-Weave-Total-Bytes This header may be included with a POST request using the batch query parameter, to indicate the total combined size of payloads in the batch, in bytes. If the server would not accept a batch containing that many bytes, then a 400 Bad Request response will be returned with response code 17. If the value of this header is not a valid positive integer value, or if the request is not operating on a batch, then a 400 Bad Request response will be returned with response code 1.
Response Headers
Retry-After When sent together with an HTTP 503 status code, this header signifies that the server is undergoing maintenance. The client should not attempt any further requests to the server for the number of seconds specified in the header value. When sent together with a HTTP 409 status code, this header gives the time after which the conflicting edits are expected to complete. Clients should wait until at least this time before retrying the request.
40 Chapter 2. Services Mozilla Services Documentation, Release
X-Weave-Backoff This header may be sent to indicate that the server is under heavy load but is still capable of servicing requests. Unlike the Retry-After header, X-Weave-Backoff may be included with any type of response, including a 200 OK. Clients should perform the minimum number of additional requests required to maintain consistency of their stored data, then not attempt any further requests for the number of seconds specified in the header value. X-Last-Modified This header gives the last-modified time of the target resource as seen during processing of the request, and will be included in all success responses (200, 201, 204). When given in response to a write request, this will be equal to the server’s current time and to the new last-modified time of any BSOs created or changed by the request. It is similar to the standard HTTP Last-Modified header, but the value is a decimal timestamp rather than a HTTP-format date. X-Weave-Timestamp This header will be sent back with all responses, indicating the current timestamp on the server. When given in response to a write request, this will be equal to the new timestamp value of any BSOs created or changed by that request. It is similar to the standard HTTP Date header, but the value is a decimal timestamp rather than a HTTP- format date. X-Weave-Records This header may be sent back with multi-record responses, to indicate the total number of records included in the response. X-Weave-Next-Offset This header may be sent back with multi-record responses where the request included a limit parameter. Its presence indicates that the number of available records exceeded the given limit. The value from this header can be passed back in the offset parameter to retrieve additional records. The value of this header will always be a string of characters from the urlsafe-base64 alphabet. The specific contents of the string are an implementation detail of the server, so clients should treat it as an opaque token. X-Weave-Quota-Remaining This header may be returned in response to write requests, indicating the amount of storage space remain- ing for the user (in KB). It will not be returned if quotas are not enabled on the server. X-Weave-Alert This header may be returned in response to any request, and contains potential warning messages, infor- mation, or other alerts. If the first character of the header is not “{” then it is intended to be a human-readable string that may be included in logs. If the first character of the header is “{” then it is a JSON object signalling impending shutdown of the service. It will contain the following fields: • code: one of the strings “soft-eol” or “hard-eol”. • message: a human-readable message that may be included in logs. • url: a URL at which more information is available.
2.2. Storage Service 41 Mozilla Services Documentation, Release
HTTP status codes
Since the syncstorage protocol is implemented on top of HTTP, clients should be prepared to deal gracefully with any valid HTTP response. This section serves to highlight the response codes that explicitly form part of the syncstorage protocol. 200 OK The request was processed successfully, and the server is returning useful information in the response body. 304 Not Modified For requests that include the X-If-Modified-Since header, this response code indicates that the resource has not been modified. The client should continue to use its local copy of the data. 400 Bad Request The request itself or the data supplied along with the request is invalid and could not be processed by the server. For example, this response will be returned if a header value is incorrectly formatted or if a JSON request body cannot be parsed. If the response has a Content-Type of application/json then the body will be an integer response code as documented in Response codes. The respcodes with particular meaning in this protocol include: • 6: JSON parse failure, likely due to badly-formed POST data. • 8: invalid BSO, likely due to badly-formed POST data. • 13: invalid collection, likely invalid chars incollection name. • 14: user has exceeded their storage quota. • 16: client is known to be incompatible with the server. • 17: server limit exceeded, likely due to too many items or too large a payload in a POST request. 401 Unauthorized The authentication credentials are invalid on this node. This may be caused by a node reassignment or by an expired/invalid auth token. The client should check with the tokenserver whether the user’s endpoint URL has changed. If it has changed, the current sync is to be aborted and should be retried against the new endpoint URL. 404 Not Found The requested resource could not be found. This may be returned for GET and DELETE requests on non-existent items. Non-existent collections do not trigger a 404 Not Found for backwards-compatibility reasons. 405 Method Not Allowed The request URL does not support the specific request method. For example, attempting a PUT request to /info/quota would produce a 405 response. 409 Conflict The write request (PUT, POST, DELETE) has been rejected due conflicting changes made by another client, either to the target resource itself or to a related resource. The server cannot currently complete the request without violating its consistency guarantees. The client should retry the request after accounting for any changes introduced by other clients. This response may include a Retry-After header indicating the time after which the conflicting edits are expected to complete. If present, clients should wait at least this many seconds before retrying the request.
42 Chapter 2. Services Mozilla Services Documentation, Release
412 Precondition Failed For requests that included the X-If-Unmodified-Since header, this response code indicates that the resource has in fact been modified more recently than the given time. The requested write operation will not have been performed. 413 Request Entity Too Large The body submitted with a write request (PUT, POST) was larger than the server is willing to accept. For multi-record POST requests, the client should retry by sending the records in smaller batches. 415 Unsupported Media Type The Content-Type header submitted with a write request (PUT, POST) specified a data format that is not supported by the server. 503 Service Unavailable Indicates that the server is undergoing maintenance. Such a response will include a Retry-After header, and the client should not attempt another sync for the number of seconds specified in the header value. The response body may contain a JSON string describing the server’s status or error. 513 Service Decommissioned Indicates that the service has been decommissioned, and presumably replaced with a new and better service using some as-yet-undesigned protocol. This response will include an X-Weave-Alert header whose value is a JSON object with the following fields: • code: the string “hard-eol”. • message: a human-readable message that may be included in logs. • url: a URL at which more information is available. The client should display an appropriate message to the user and cease any further attempts to use the service.
Concurrency and Conflict Management
The SyncStorage service allows multiple clients to synchronize data via a shared server without requiring inter-client coordination or blocking. To achieve proper synchronization without skipping or overwriting data, clients are expected to use timestamp-driven coordination features such as X-Last-Modified and X-If-Unmodified-Since. The server guarantees a strictly consistent and monotonically-increasing timestamp across the user’s stored data. Any request that alters the contents of a collection will cause the last-modified time to increase. Any BSOs added or modified by such a request will have their “modified” field set to the updated timestamp. Conceptually, each write request will perform the following operations as an atomic unit: • Read the current time T, and check that it’s greater than the overall last-modified time for the user’s data. If not then return a 409 Conflict. • Create any new BSOs as specified by the request, setting their “modified” field to T. • Modify any existing BSOs as specified by the request, setting their “modified” field to T. • Delete any BSOs as specified by the request. • Set the last-modified time for the collection to T. • Set the overall last-modified time for the user’s data to T. • Generate a 200 OK response with the X-Last-Modified and X-Weave-Timestamp headers set to T.
2.2. Storage Service 43 Mozilla Services Documentation, Release
While write requests from different clients may be processed concurrently by the server, they will appear to the clients to have occurred sequentially, instantaneously and atomically according to the above sequence. To avoid having the server transmit data that has not changed since the last request, clients should set the X-If- Modified-Since header and/or the newer parameter to the last known value of X-Last-Modified on the target resource. To avoid overwriting changes made by others, clients should set the X-If-Unmodified-Since header to the last known value of X-Last-Modified on the target resource.
Examples
Example: polling for changes to a BSO
To efficiently check for changes to an individual BSO, use GET /storage/
last_modified=0 while True: headers={"X-If-Modified-Since": last_modified} r= server.get("/collection/id", headers) if r.status != 304: print" MODIFIED ITEM:", r.json_body last_modified=r.headers["X-Last-Modified"]
Example: polling for changes to a collection
To efficiently poll the server for changes within a collection, use GET /storage/
last_modified=0 while True: r= server.get("/collection?newer="+ last_modified) for item in r.json_body["items"]: print"MODIFIED ITEM:", item last_modified=r.headers["X-Last-Modified"]
Example: safely updating items in a collection
To update items in a collection without overwriting any changes made by other clients, use POST /stor- age/
r= server.get("/collection") last_modified=r.headers["X-Last-Modified"]
bsos= generate_changes_to_the_collection()
headers={"X-If-Unmodified-Since": last_modified} r= server.post("/collection", bsos, headers)
44 Chapter 2. Services Mozilla Services Documentation, Release
if r.status == 412: print"WRITE FAILED DUE TO CONCURRENT EDITS"
The client may choose to abort the write, or to merge the changes from the server and re-try with an updated value of X-Last-Modified. A similar technique can be used to safely update a single BSO using PUT /storage/
Example: creating a BSO only if it does not exist
To specify that a BSO should be created only if it does not already exist, use the X-If-Unmodified-Since header with the special value of 0:
headers={"X-If-Unmodified-Since":"0"} r= server.put("/collection/item", data, headers) if r.status == 412: print"ITEM ALREADY EXISTS"
Example: paging through a large set of items
The syncstorage server allows efficient paging through a large set of items by using the limit and offset parameters. Clients should begin by issuing a GET /storage/
r= server.get("/collection?limit=100") print"GOT ITEMS:", r.json_body["items"]
last_modified=r.headers["X-Last-Modified"] next_offset=r.headers.get("X-Weave-Next-Offset")
while next_offset: headers={"X-If-Unmodified-Since": last_modified} r= server.get("/collection?limit=100&offset="+ next_offset, headers)
if r.status == 412: print"COLLECTION WAS MODIFIED WHILE READING ITEMS" break
print"GOT ITEMS:", r.json_body["items"] next_offset=r.headers.get("X-Weave-Next-Offset")
2.2. Storage Service 45 Mozilla Services Documentation, Release
Example: uploading a large batch of items
The syncstorage server allows several upload requests to be combined into a single “batch” so that they all become visible to other clients as a single atomic unit. This is achieved by using the batch and commit parameters on the upload request. Clients should begin by issuing a POST /storage/
# Make an initial request to start a batch upload. # It's possible to send some items here, but not required. r= server.post("/collection?batch=true", []) # Note that the batch id is opaque and cannot be safely put in a URL directly batch_id= urllib.quote(r.json_body["batch"])
# Always use X-If-Unmodified-Since to detect conflicts. last_modified=r.headers["X-Last-Modified"] headers={"X-If-Unmodified-Since": last_modified}
for items in split_items_into_smaller_batches():
# Send the items in several smaller batches. r= server.post("/collection?batch="+ batch_id, items, headers) if r.status == 412: raise Exception("COLLECTION WAS MODIFIED WHILE UPLOADING ITEMS")
# The collection will not be modified yet. assert r.headers['X-Last-Modified'] == last_modified
# Commit the batch once all items are uploaded. # Again, it's possible to send some final items here, but not required. r= server.post("/collection?commit=true&batch="+ batch_id, [], headers) if r.status == 412: raise Exception("COLLECTION WAS MODIFIED WHILE COMMITTING ITEMS")
# At this point all the uploaded items become visible, # and the collection appears modified to other clients. assert r.headers['X-Last-Modified']> last_modified
Changes from v1.1
The following is a summary of protocol changes from Storage API v1.1 along with a justification for each change:
46 Chapter 2. Services Mozilla Services Documentation, Release
What Changed Why Authentication is now performed using a This supports authentication via Firefox Accounts and allows us to BrowserID-based tokenserver flow and iterate the details of that flow without changing the sync protocol. HAWK Access Authentication. The structure of the endpoint URL is no This was unnecessary coupling and clients do not need to longer specified, and should be considered change/configure components of the endpoint URL. URL handling an implementation detail. must change already to support TokenServer-based authentication. The datatypes and defaults of BSO fields are This reflects current server behavior, and seems prudent to specify more precisely specified. more explicitly. The BSO fields “parentid” and These were deprecated in version 1.1 and are not in active use in “predecessorid” have been removed along current versions of Firefox. with any related query parameters. The ‘application/whoisi’ output format has This is not used in any current versions of Firefox. been removed. The previously-undocumented This actually is used so we better document it. X-Weave-Quota-Remaining header has been documented. The X-Confirm-Delete header has been This is sent unconditionally by current client code, and is removed. therefore useless. Existing client code can safely continue to send it, and it will be ignored by the server. The X-Weave-Alert header has grown This is already implemented in current Firefox so we better additional semantics related to service document it. end-of-life announcements. GET /storage/collection no longer accepts These are not in active use in current versions of Firefox, and ‘index_above’ or ‘index_below’ impose additional requirements on the server that may limit operational flexibility. DELETE /storage/collection no longer These are not in active use in current versions of Firefox, are not accepts query parameters other than ‘ids’ all implemented correctly in the current server, and impose additional requirements on the server that may limit operational flexibility. POST /storage/collection now accepts This matches nicely with ‘application/newlines’ as supported ‘application/newlines’ input in addition to already in response bodies, and may enable more efficient request ‘application/json’. streaming in future. Existing client code doesn’t need to change. The offset parameter is now an opaque The parameter is not in active use in current versions of Firefox, server-generated value, and clients must not and its existing semantics are difficult to implement efficiently on create their own values for it. the server. This change allows for more efficient pagination of results in future client code. The X-Last-Modified header has been added. This has slightly different semantics to the X-Weave-Timestamp header and may be used by future clients for better conflict management. Existing client code doesn’t need to change. The X-If-Modified-Since header has been Existing client code doesn’t need to change, but will allow future added and can be used on all GET requests. client code to avoid transmission of redundant data. The X-If-Unmodified-Since header can be Existing client code doesn’t need to change, but will allow future used on some GET request. client code to detect changes during paginated fetching of results. The server may reject concurrent write This will be visible to existing client code, but can be handled like attempts with a 409 Conflict. a 503 error. It lets the server provide much stronger consistency guarantees that will improve overall robustness of the service. Batch uploads are supported that cross This is a backwards-compatible API extension that allows clients several POST requests. to ensure consistency of their uploaded items. Various server-specific size limits can be This is a backwards-compatible API extension that allows clients read from a new /info/configuration to ensure interoperability with configurable server behaviour. endpoint.
2.2. Storage Service 47 Mozilla Services Documentation, Release
History
XXX
Resources
• Server (v1.1): https://hg.mozilla.org/services/server-storage • Server (v1.5): https://github.com/mozilla-services/server-syncstorage • Client: Firefox 4, Firefox Home
Registration
Goal of the Service
The reg server provides web services on the top of the authentication back-end that can be used to: • create or delete a new user • change a user password or e-mail • assign/get a storage node for a user • return an HTML reCaptcha challenge It’s currently used by all Sync clients applications. If you are using our hosted services with your own client code, please read the Term of Services.
Documentation content
User API v1.0
General description
The URL for a Reg request is structured as follows: https://
Component Mozilla Default Description server name auth.services.mozilla.com the hostname of the server pathname user the prefix associated with the service on the box version 1.0 The API version. username (none) the name of the object (user) to be manipulated further (none) The additional function information as defined in the paths instruction below Certain functions use HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not match the username in the path, the server will issue an Error Response. The Sync User API has a set of Response codes to cover errors in the request or on the server side. The format of a successful response is defined in the appropriate request method section.
48 Chapter 2. Services Mozilla Services Documentation, Release
APIs
GET https://server/pathname/version/username Returns 1 if the username is in use, 0 if it is available. The answer is in plain text. Possible errors: • 503: there was an error getting the information GET https://server/pathname/version/username/node/weave Returns the Weave (aka Sync) Node that the client is located on. Sync-specific calls should be directed to that node. Return value: the node URL, an unadorned (not JSON) string. node may be ‘null’ if no node can be assigned at this time, probably due to sign up throttling. Possible errors: • 503: there was an error getting a node | empty body • 404: user not found | empty body GET https://server/pathname/version/username/password_reset Requests a password reset email be mailed to the email address on file. Returns ‘success’ if an email was successfully sent. If captchas are enabled for the site, requires captcha-challenge and captcha-response parameters. Return value: • “success” Possible errors: • 503: problems with looking up the user or sending the email • 400: 12 (No email address on file) • 400: 3 (Incorrect or missing username) • 400: 2 (Incorrect or missing captcha) PUT https://server/pathname/version/username Requests that an account be created for username. The body is a JSON mapping and should include: • password: the password to be associated with the account. • e-mail: Email address associated with the account. • captcha-challenge: The challenge string from the captcha. • captcha-response: The response to the captcha. An X-Weave-Secret can be provided containing a secret string known by the server. When provided, it will override the captcha. This is useful for testing and automation. The server will return the lowercase username on success. Possible errors: • 503: there was an error creating the reset code • 400: 4 (user already exists)
2.3. Registration 49 Mozilla Services Documentation, Release
• 400: 6 (Json parse failure) • 400: 12 (No email address on file) • 400: 7 (Missing password field) • 400: 9 (Requested password not strong enough) • 400: 2 (Incorrect or missing captcha) POST https://server/pathname/version/username/password Changes the password associated with the account to the value specified in the POST body. NOTE: Requires basic authentication with the username and (current) password associated with the ac- count. The auth username must match the username in the path. Alternately, a valid X-Weave-Password-Reset header can be used, if it contains a code previously obtained from the server. Return values: “success” on success. Possible errors: • 400: 7 (Missing password field) • 400: 10 (Invalid or missing password reset code) • 400: 9 (Requested password not strong enough) • 404: the user does not exists in the database • 503: there was an error updating the password • 401: authentication failed POST https://server/pathname/version/username/email Changes the email associated with the account to the value specified in the POST body. NOTE: Requires basic authentication with the username and password associated with the account. The auth username must match the username in the path. Alternately, a valid X-Weave-Password-Reset header can be used, if it contains a code previously obtained from the server. Return values: The user email on success. Possible errors: • 400: 12 (No email address on file) • 404: the user does not exists in the database • 503: there was an error updating the email • 401: authentication failed DELETE https://server/pathname/version/username Deletes the user account. NOTE: Requires simple authentication with the username and password associated with the account. The auth username must match the username in the path. Return value: • 0 on success Possible errors:
50 Chapter 2. Services Mozilla Services Documentation, Release
• 503: there was an error removing the user • 404: the user does not exist in the database • 401: authentication failed GET https://server/misc/1.0/captcha_html Returns an html body string containing a reCaptcha challenge captcha. The PUT API to create a user will expect the challenge and response from this captcha. Note: this function outputs html, not json.
X-Weave-Alert
This header may be sent back from any transaction, and contains potential warning messages, information, or other alerts. The contents are intended to be human-readable.
History
XXX
Resources
• Original wiki page: https://wiki.mozilla.org/Services/Sync/Server/API/User/1.0 • Continuous Integration server: XXX • Server: https://hg.mozilla.org/services/server-reg • Client: Firefox 4+
Easy Setup
Goal of the Service
Setting up a new mobile device should only involve entering a short code on the desktop device. Secondary request, not a hard requirement, is that if the user has a mobile device, and is setting up a desktop device, that the flow is similar and still involves entering the key on the desktop. If you are using our hosted services with your own client code, please read the Term of Services.
Documentation content
User Flow
Desired User Flow
There are two possible scenarios when pairing two devices. In the simplest case, one of the devices is already connected to Sync. This is the only scenario considered in v1/v2. In other other scenario, none of the devices are connected to Sync and the user must connect to or create a Sync account on one of them after the pairing. This scenario is new in v3.
2.4. Easy Setup 51 Mozilla Services Documentation, Release
For the sake of this document, “Mobile” refers to a mobile device or a desktop computer, and “Desktop” refers to a desktop computer. 1. User chooses “Pair Device” on Mobile. 2. Mobile displays a setup key that contains both the initial secret and a channel ID. 3. On Desktop, user chooses “Pair Device” according to the instructions displayed on the device. 4. Mobile and Desktop exchange messages to build the secure channel. 5. Once the channel has been established and Mobile and Desktop are “paired”, the user creates a Sync account on Desktop and Desktop uploads its data to the server during the first sync. This step is not present in v1/v2 of this flow. It can be omitted if the user already has an account. 6. Desktop transmits the Sync account credentials (username, password, Sync Key) to Mobile via the channel. 7. Mobile completes setup and starts syncing.
Overview
• Mobile and Desktop complete the two roundtrips of J-PAKE messages to agree upon a strong secret K. • A 256-bit key is derived from K using HMAC-SHA256 using a fixed extraction key. • The encryption and HMAC keys are derived from that 256-bit key using HMAC-SHA256. • To establish the pairing, Mobile encrypts the known message “0123456789ABCDEF” with the AES key and uploads it. Desktop verifies that it has the same key by encrypting the known message with the key that Desktop derived. • To exchange credentials after a successful pairing and possibly account creation on Desktop: – Desktop encrypts the credentials with the encryption key and uploads the encrypted credentials in turn, adding a HMAC-SHA256 hash of the ciphertext (using the HMAC key). – Mobile verifies the ciphertext against the HMAC-SHA256 hash. If successful, Mobile decrypts ciphertext and applies credentials – At this point, Mobile can begin synchronizing. Mobile Server Desktop ======| retrieve channel<------| generate random secret| show PIN= secret+ channel| ask user for PIN upload Mobile's message 1 ------>| |----> retrieve Mobile's message 1 |<----- upload Desktop's message 1 retrieve Desktop's message 1 <---| upload Mobile's message 2 ------>| |----> retrieve Mobile's message 2 | compute key |<----- upload Desktop's message 2 retrieve Desktop's message 2 <---| compute key| encrypt known value------>| |------> retrieve encrypted value | verify against local known value
At this point Desktop knows whether the PIN was entered correctly.
52 Chapter 2. Services Mozilla Services Documentation, Release
If it wasn't, Desktop deletes the session. If it was, the account setup can proceed. If Desktop doesn't have an account set up yet, it will keep the channel open and let the user connect to or create an account.
| encrypt credentials |<------upload credentials retrieve credentials<------| verify HMAC| decrypt credentials| delete session------>| start syncing|
Detailed Flow
1. Mobile asks server for new channel ID (4 characters a-z0-9)
C: GET/new_channel HTTP/1.1 S:"a7id"
2. Mobile generates PIN from random weak secret (4 characters a-z0-9) and the channel ID, computes and uploads J-PAKE msg 1. New in v2: To prevent double uploads in case of retries, the If-None-Match: * header may be specified. This ensures that the message is only uploaded if the channel is empty. If it is not then the request will fail with a 412 Precondition Failed which should be considered the same as 200 OK. The 412 will also contain the Etag of the data was the client just uploaded.
C: PUT/a7id HTTP/1.1 C: If-None-Match: * C: C: { C:'type':'receiver1', C:'version':3,// new in v3 C:'payload':{ C:'gx1':'45...9b', C:'zkp_x1':{ C:'b':'09e22607ead737150b1a6e528d0c589cb6faa54a', C:'gr':'58...7a' C:'id':'receiver', C: } C:'gx2':'be...93', C:'zkp_x2':{ C:'b':'222069aabbc777dc988abcc56547cd944f056b4c', C:'gr':'5c...23' C:'id':'receiver', C: } C: } C: }
Success response:
S: HTTP/1.1 200OK S: ETag:"etag-of-receiver1-message"
New in v2: Response that will be returned on retries if the Desktop already replaced the message:
2.4. Easy Setup 53 Mozilla Services Documentation, Release
S: HTTP/1.1 412 Precondition Failed S: ETag:"etag-of-receiver1-message"
3. Desktop asks user for the PIN, extracts channel ID and weak secret, fetches Mobile’s msg 1:
C: GET/a7id HTTP/1.1
Success response:
S: HTTP/1.1 200OK S: ETag:"etag-of-receiver1-message"
New in v3: Prior to v3, clients would only allow a 10 second timeout for messages after the first. This means that if Desktop does not yet have credentials, a Mobile client that implements v2 or lower will not wait for the account setup to finish. Desktop should therefore detect Mobile’s API version at this point and abort the pairing right away if there are no credentials present on Desktop. 4. Desktop computes and uploads msg 1. New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means that our first PUT was actually correctly received by the server and that the other side has already uploaded its next message. So just consider the 412 to be a 200.
C: PUT/a7id HTTP/1.1 C: If-Match:"etag-of-receiver1-message" C: C: { C:'type':'sender1', C:'version':3,// new in v3 C:'payload':{ C:'gx1':'45...9b', C:'zkp_x1':{ C:'b':'09e22607ead737150b1a6e528d0c589cb6faa54a', C:'gr':'58...7a' C:'id':'sender', C: } C:'gx2':'be...93', C:'zkp_x2':{ C:'b':'222069aabbc777dc988abcc56547cd944f056b4c', C:'gr':'5c...23' C:'id':'sender', C: } C: } C: }
Success response:
S: HTTP/1.1 200OK S: Etag:"etag-of-sender1-message"
New in v2: Response that will be returned on retries if Mobile already replaced the message:
S: HTTP/1.1 412 Precondition Failed S: Etag:"etag-of-sender1-message"
5. Mobile polls for Desktop’s msg 1 once per second for at least 300 seconds:
54 Chapter 2. Services Mozilla Services Documentation, Release
C: GET/a7id HTTP/1.1 C: If-None-Match:"etag-of-receiver1-message"
S: HTTP/1.1 304 Not Modified
Mobile tries again after 1 second:
C: GET/a7id HTTP/1.1
S: HTTP/1.1 200OK S: Etag:"etag-of-sender1-message"
Mobile computes and uploads msg 2. New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means that our first PUT was actually correctly received by the server and that the other side has already uploaded its next message. In this instance, the client can effectively consider the 412 to be a 200.:
C: PUT/a7id HTTP/1.1 C: If-Match:"etag-of-sender1-message" C: C: { C:'type':'receiver2', C:'version':3,// new in v3 C:'payload':{ C:'A':'87...82', C:'zkp_A':{ C:'b':'6f...08', C:'id':'receiver', C:'gr':'f8...49' C: } C: } C: }
S: HTTP/1.1 200OK S: ETag:"etag-of-receiver2-message"
New in v2: Response that will be returned on retries if Desktop already replaced the message:
S: HTTP/1.1 412 Precondition Failed S: ETag:"etag-of-receiver2-message"
6. Desktop polls for Mobile’s msg 2 once per second for at least 10 seconds:
C: GET/a7id HTTP/1.1 C: If-None-Match:"etag-of-sender1-message"
S: HTTP/1.1 304 Not Modified
and eventually retrieves it:
S: HTTP/1.1 200OK S: Etag:"etag-of-receiver2-message"
Desktop computes key, computes and uploads msg 2. New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous
2.4. Easy Setup 55 Mozilla Services Documentation, Release
message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means that our first PUT was actually correctly received by the server and that the other side has already uploaded its next message. In this instance, the client can effectively consider the 412 to be a 200.
C: PUT/a7id HTTP/1.1 C: If-Match:"etag-of-receiver2-message" C: C: { C:'type':'sender2', C:'version':3,// new in v3 C:'payload':{ C:'A':'87...82', C:'zkp_A':{ C:'b':'6f...08', C:'id':'sender', C:'gr':'f8...49' C: } C: } C: }
S: HTTP/1.1 200OK S: ETag:"etag-of-sender2-message"
New in v2: Response that will be returned on retries if Mobile already replaced the message:
S: HTTP/1.1 412 Precondition Failed S: ETag:"etag-of-sender2-message"
7. Mobile polls for Desktop’s msg 2 once per second for at least 10 seconds and eventually retrieves it:
C: GET/a7id HTTP/1.1 C: If-No-Match:"etag-of-receiver2-message"
S: HTTP/1.1 200OK S: Etag:"etag-of-sender2-message" {'type':'sender2',...}
S: HTTP/1.1 304 Not Modified
Mobile computes key, uploads encrypted known message “0123456789ABCDEF” to prove its knowledge (msg 3). New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means that our first PUT was actually correctly received by the server and that the other side has already uploaded its next message. In this instance, the client can effectively consider the 412 to be a 200.
C: PUT/a7id HTTP/1.1 C: If-Match:"etag-of-sender2-message" C: C: { C:'type':'receiver3', C:'version':3,// new in v3 C:'payload':{ C:'ciphertext':"base64encoded=", C:'IV':"base64encoded=", C: } C: }
56 Chapter 2. Services Mozilla Services Documentation, Release
S: HTTP/1.1 200OK S: Etag:"etag-of-receiver3-message"
New in v2: Response that will be returned on retries if Desktop already replaced the message:
S: HTTP/1.1 412 Precondition failed S: Etag:"etag-of-receiver3-message"
8. Desktop retrieves Mobile’s msg 3 to confirm the key. It polls once per second for at least 10 seconds:
C: GET/a7id HTTP/1.1 C: If-No-Match:"etag-of-sender2-message"
S: HTTP/1.1 200OK C: ETag:"etag-of-receiver3-message" ...
Desktop verifies it against its own version. If the values don’t match, the pairing is aborted and the session should be deleted. Once credentials are available, and if the channel is still available, Desktop encrypts the credentials and uploads them. New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means that our first PUT was actually correctly received by the server and that the other side has already uploaded its next message. In this instance, the client can effectively consider the 412 to be a 200. New in v3: Desktop must include the If-Match header to ensure the session hasn’t been deleted yet (e.g., due to a timeout) or tampered with in the meantime.
C: PUT/a7id HTTP/1.1 C: If-Match:"etag-of-receiver3-message" C: C: { C:'type':'sender3', C:'version':3,// new in v3 C:'payload':{ C:'ciphertext':"base64encoded=", C:'IV':"base64encoded=", C:'hmac':"base64encoded=", C: } C: }
S: HTTP/1.1 200OK S: Etag:"etag-of-sender3-message"
New in v2: Response that will be returned on retries if Mobile already replaced the message:
S: HTTP/1.1 412 Precondition failed S: Etag:"etag-of-sender3-message"
If the hash does not match, the Desktop deletes the session:
C: DELETE/a7id HTTP/1.1
2.4. Easy Setup 57 Mozilla Services Documentation, Release
S: HTTP/1.1 200OK ...
This means that Mobile will receive a 404 when it tries to retrieve the encrypted credentials. 9. Mobile polls for the encrypted credentials once per second for at least 300 seconds to allow for the account process (the increased timeout is new in v3):
C: GET/a7id HTTP/1.1 C: If-None-Match:"etag-of-receiver3-message"
S: HTTP/1.1 200OK ...
Decrypts Sync credentials and verifies HMAC. 10. Mobile deletes the session [OPTIONAL]
C: DELETE/a7id HTTP/1.1
S: HTTP/1.1 200OK ...
11. Mobile starts syncing.
Server API v1.0
General Description
The only valid HTTP response codes are 200 and 304 since those are part of the protocol and expected to happen. Anything else, like 400, 403, 404 or 503 must result in a complete termination of the password exchange. The client can retry the exchange then at a later time, starting all over with clean state. Every call must be done with a ‘’X-KeyExchange-Id” header, containing a half-session identifier for the channel. This client ID must be a string of 256 chars. The server will keep track of the two first ids used for a given channel, from its creation to its deletion and will close the channel and issue a 400 if any request is made with an unknown id or with no id at all. Last, if a given IP attempts to flood the server with a lot of calls in a short time, it will be blacklisted for 10 minutes return 403s in the interim for any requests made from the same IP. When receiving this error code, legitimate clients can fall back to a manual transaction. A client that generates a lot of bad requests will also be blacklisted, but for an hour.
APIS
GET https://server/new_channel Returns in the response body a JSON-encoded random channel id of N chars from [a-z0-9]. When the API is called, the id returned is guaranteed to be unique. The created channel will have a limited TTL (currently configured to 5 minutes). Return codes: • 200: channel created successfully • 503: the server was unable to create a new channel.
58 Chapter 2. Services Mozilla Services Documentation, Release
• 400: Bad or no client ID. The channel is deleted. • 403: the IP is blacklisted. GET https://server/channel_id Returns in the response body the content of the channel of id ‘’channel_id’‘. Returns an ‘’ETag” response header containing a unique hash. The request can contain a ‘’If-None-Match” header containing a hash. If the hash is similar to the current hash of the channel, the server will return a 304 and an empty body. The number of GET calls for a given channel are limited to 6. The channel will be deleted by the server after 6 successful GETs. Return codes: • 200: data retrieved successfully • 404: the channel does not exists. It was not created by a call to ‘’new_channel” or timed out. • 304: the data was not changed. • 400: Bad or no client ID. The channel is deleted. • 403: the IP is blacklisted. PUT https://server/channel_id Put in the channel of id ‘’channel_id” the content of the request body. Returns an ‘’ETag” response header containing a unique hash. If an If-Match Header is provided, it must be the value of the etag before the update to the channel is applied, or *. If different, a precondition failed code is returned. If a If-None-Match header is provided and equals to *, and if the channel is not empty (e.g. some data has been already put in the channel), a precondition failed code is returned. Return codes: • 200: data set successfully • 404: the channel does not exists. It was not created by a call to ‘’new_channel” or timed out. • 400: Bad or no client ID. The channel is deleted. • 403: the IP is blacklisted. • 412: a precondition failed. POST https://server/report Reports a log to the server, and optionally asks for a channel deletion. The log is the body of the request. If the request contains a ‘’X-KeyExchange-Log” header, its value is prepended to the log provided in the body. In other words, the header can be used for small logs, and the body for more info. The body size is limited to 2000 chars. If both body and headers are empty, nothing is logged. The current errors reported by the client are described in the next section, but the log is a free-form string. Warning: Under a normal exchange the server is able to count the number of calls and close the channel at the end, so this API is not to be used to close the channel. Some value should be reported to generate a security log. The client is therefore encouraged to always provide a report value when calling. Optionally, if the request contains the ‘’X-KeyExchange-Id” header and a ‘’X-KeyExchange-Cid” header containing the channel id, the channel will be deleted by the server. Return codes:
2.4. Easy Setup 59 Mozilla Services Documentation, Release
• 200: logged successfully • 403: the IP is blacklisted. • 400: bad request (missing log or bad ids)
Error messages
• jpake.error.timeout (Timeout) : Reported when the exchange is aborted due to timeouts. • jpake.error.invalid (Invalid message): Reported when a malformed message is received. A malformed message is one that doesn’t correctly parse as JSON. • jpake.error.wrongmessage (Wrong message): Reported when the wrong message is received, as identified by the type property in the JSON blob. • jpake.error.internal (Internal J-PAKE failure): Reported when a J-PAKE computation step or encryp- tion/decryption step fails. • jpake.error.keymismatch (Key mismatch): Reported when the SHA256 or HMAC verification fails, in other words when the PIN wasn’t entered correctly and both sides ended up with different keys. • jpake.error.server (Unexpected Server Response): Reported when unexpected HTTP response from the J- PAKE server is received. • jpake.error.userabort (User Abort) : Reported when a client aborts the J-PAKE transaction; for example, when canceling a setup wizard.
Security
DOS Defense
• Least Recently Used (LRU) queue approach for monitoring IP addresses issuing frequent requests • Configurable threshold for adding IP address to Blacklist/Penalty Box • Configurable time-out for IP addresses added to Blacklist/Penalty Box • A single shared blacklist will exist within memcache • LRU queues will be unique to each server and will penalize an IP to the shared blacklist on memcache • All thresholds will be controlled via the configuration page
TearDown DOS Defense
• Tear down requires valid channel and valid x-keyexchange-id value • Statistically unlikely. Channel is 4 characters and keyexchange-id is 255 characters • Brute force attempts will generate lots of noise and will be limited per DOS defense
Logging Points
CEF Logging
• Bad action taken against a valid channel id (denoted by 400 error code)
60 Chapter 2. Services Mozilla Services Documentation, Release
• Examples: non-existent x-keyexchange-id, bad x-keyexchange-id • Action taken against an invalid channel id • Examples: request for properly formed, but not existing, channel id • IP address sent to black list due to DOS prevention controls • Examples: Flood of requests from a single IP • Client fallback to original sync method • Examples: Client unable to complete J-PAKE sync for any number of reasons and falls back to original sync approach • Reported by client to server via reporting API
Application Logging
• Full application logging will be created to enable incident response review • Logged to application server and not via CEF • Logs will include: – Timestamp – IP address – Full URL – x-keyexchange-id – Event – Other non-essential headers will be discarded
Admin Web Page
• A small web administrator page will be created which will allow an admin to view all IP addresses that are currently blacklisted. • The admin will be able to unblock any of the IP addresses through this page • Otherwise the IP address will be removed from the black list after the time has elapsed that is defined within the configuration file • Access to the web page will be password-protected with a simple .htaccess file and IP filtering access (10.*.*.*)
History v3 - 2012-01-26
• Added a version attribute to the JSON payloads. • Requiring longer timeout for the final message (credentials exchange) to allow for the account creation flow on Desktop after pairing devices.
2.4. Easy Setup 61 Mozilla Services Documentation, Release v2 - 2011-04-26
• Added support for If-None-Match and If-Match on PUT requests and the corresponding 412 Precondition Failed response code to improve reliability on flaky networks. v1
• Initial implementation
Resources
• Original wiki design page/meeting notes: https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE • Continuous Integration server: XXX • Server: https://hg.mozilla.org/services/server-key-exchange • Clients – Firefox Home: https://hg.mozilla.org/services/fx-home – Firefox: XXX define a link to a search to display all related files in mozilla-central – Firefox Mobile: XXX define a link to a search to display all related files in mobile-browser
Secure-Registration (Mozilla specific)
Warning: The Sreg service is used internally in the Mozilla infrastructure for security reasons and it is most likely that you will never need it outside Mozilla.
Goal of the Service
The Sreg server provides web services on the top of the authentication back-end that can be used to: • create a new user • delete a user • change a user password • e-mail password reset codes • delete password reset codes It’s used by the mozilla auth backend in server-core in order to separate system writes that can be done with the users credentials from those that require ldap admin credentials. This provides an additional level of security by keeping the auth credentials only on a box with limited access. The mozilla backend is used by the Authentication server and the Account manager server. Note that some write operations (2) are not delegated, like changing the DN & the e-mail address of the user, so the Account manager will still bind for writes as the user. Sreg does not require any authentication, as it is not intended to be used without first going through a primary gate- way that performs any necessary authentication before proxying the requests. The server must remain private to our infrastructure with as little outside access as possible
62 Chapter 2. Services Mozilla Services Documentation, Release
Documentation content
Server API v1.0
General description
The URL for a Sreg request is structured as follows: http://
The Sync User API has a set of Response codes to cover errors in the request or on the server side. The format of a successful response is defined in the appropriate request method section.
APIs
GET https://server/username/node/weave Gets the user a weave node. In general, if the Sreg version is called, it is expected that a new node will be assigned (client should already have failed to get the node out of ldap), but this is not necessarily the case, and it can return a currently assigned node. The plan is to abstract this call into a separate api that handles node assignment, but it is noted here to make sure we remain backwards compatible until that work is complete. Return value:
2.5. Secure-Registration (Mozilla specific) 63 Mozilla Services Documentation, Release
node may be ‘null’ if no node can be assigned at this time. Note that the returned value is a json-encoded value, so strings are quoted. Possible errors: • 503: there was an error getting a node | empty body • 404: user not found | empty body GET https://server/username/password_reset_code Generates a reset code and mails it to the email address associated with
64 Chapter 2. Services Mozilla Services Documentation, Release
• 404: the user does not exists in the database • 503: there was an error updating the password (including a potentially bad reset code) DELETE https://server/username/password_reset_code Removes a password reset code, if any exists. Return value: • 0 on success Possible errors: • 503: there was an error removing the password reset code • 404: the user does not exist in the database DELETE https://server/username Delete the user’s account. The body is a JSON mapping and should include: • password: The user’s password for confirmation. Return Value: • 0 on success • 400: 7 (Missing password field) • 400: 6 (Json parse failure) • 404: the user does not exist in the database • 503: there was an error removing the user (including a potential bad password)
History
XXX
Resources
• Original wiki design page/meeting notes: https://wiki.mozilla.org/Services/Sync/SRegAPI • Continuous Integration server: XXX • Server: https://hg.mozilla.org/services/server-sreg • Client: https://hg.mozilla.org/services/server-core/file/tip/services/auth/mozilla_sreg.py
Token Server
Goal of the Service
So here’s the challenge we face. Current login for sync looks like this: 1. provide username and password 2. we log into ldap with that username and password and grab your sync node
2.6. Token Server 65 Mozilla Services Documentation, Release
3. we check the sync node against the url you’ve accessed, and use that to configure where your data is stored. This solution works great for centralized login. It’s fast, has a minimum number of steps, and caches the data centrally. The system that does node-assignment is lightweight, since the client and server both cache the result, and has support for multiple applications with the /node/
Assumptions
• A Login Server detains the secret for all the Service Nodes for a given Service. • Any given webhead in a cluster can receive calls to all service nodes in the cluster. • The Login Server will support only BrowserID at first, but could support any authentication protocol in the future, as long as it can be done with a single call • All servers are time-synced • The expires value for a token is a fixed value per application. For example it could be 30 minutes for Sync and 2 hours for bipostal. • The Login Server keeps a white list of domains for BID verifications
Documentation content
Token Server API v1.0
Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types.
GET /1.0/
66 Chapter 2. Services Mozilla Services Documentation, Release
• app_version is the specific version number of the api that you want to access. The first /1.0/ in the URL defines the version of the authentication token itself. Example for BrowserID:
GET/1.0/sync/1.5 Host: token.services.mozilla.com Authorization: BrowserID
This API returns several values in a json mapping: • id – a signed authorization token, containing the user’s id for the application and the node. • key – a secret derived from the shared secret • uid – the user id for this service • api_endpoint – the root URL for the user for the service. • duration – the validity duration of the issued token, in seconds. Example:
HTTP/1.1 200OK Content-Type: application/json
{'id':
If the X-Client-State header is included in the request, the server will compare the submitted value to any previously-seen value. If it has changed then a new uid and api_endpoint are generated, in effect “resetting” the node allocation for this user.
Request Headers
X-Client-State An optional string that can be sent to identify a unique configuration of client-side state. It may be up to 32 characters long, and must contain only characters from the urlsafe-base64 alphabet (i.e. alphanumeric characters, underscore and hyphen) and the period. A change in the value of this header will cause the user’s node allocation to be reset. Clients should include any client-side state that is necessary for accessing the selected app. For example, clients accessing SyncStorage API v1.5 would include a hex-encoded hash of the encryption key in this header, since a change in the encryption key will make any existing data unreadable. Updated values of the X-Client-State will be rejected with an error status of “invalid-client-state” if: • The proposed new value is in the server’s list of previously-seen client-state values for that user. • The client-state is missing or empty, but the server has previously seen a non-empty client-state for that user. • The user’s IdP provides generation numbers in their identity certificates, and the changed client-state value does not correspond to an increase in generation number.
2.6. Token Server 67 Mozilla Services Documentation, Release
Response Headers
Retry-After When sent together with an HTTP 503 status code, this header signifies that the server is undergoing maintenance. The client should not attempt any further requests to the server for the number of seconds specified in the header value. X-Backoff This header may be sent to indicate that the server is under heavy load but is still capable of servicing requests. Unlike the Retry-After header, X-Backoff may be included with any type of response, including a 200 OK. Clients should avoid unnecessary requests to the server for the number of seconds specified in the header value. For example, clients may avoid pre-emptively refreshing token if an X-Backoff header was recently seen. X-Timestamp This header will be included with all “200” and “401” responses, giving the current POSIX timestamp as seen by the server, in seconds. It may be useful for client to adjust their local clock when generating BrowserID assertions.
Error Responses
All errors are also returned, wherever possible, as json responses following the structure described in Cornice. In cases where generating such a response is not possible (e.g. when a request if so malformed as to be unparsable) then the resulting error response will have a Content-Type that is not application/json. The top-level JSON object in the response will always contain a key named status, which maps to a string identifying the cause of the error. Unexpected errors will have a status string of “error”; errors expected as part of the protocol flow will have a specific status string as detailed below. Error status codes and their corresponding output are: • 404 : unknown URL, or unsupported application. • 400 : malformed request. Possible causes include a missing option, bad values or malformed json. • 401 : authentication failed or protocol not supported. The response in that case will contain WWW-Authenticate headers (one per supported scheme) and may report the following status strings: – “invalid-credentials”: authentication failed due to invalid credentials e.g. a bad signature on the BrowserID assertion. – “invalid-timestamp”: authentication failed because the included timestamp differed too greatly from the server’s current time. – “invalid-generation”: authentication failed because the server has seen credentials with a more recent generation number. – “invalid-client-state”: authentication failed because the server has seen an updated value of the X-Client- State header. – “new-users-disabled”: authentication failed because the user has not been seen previously on this server, and new user accounts have been disabled in the application config. • 405 : unsupported method • 406 : unacceptable - the client asked for an Accept we don’t support
68 Chapter 2. Services Mozilla Services Documentation, Release
• 503 : service unavailable (ldap or snode backends may be down)
User Flow
Here’s the proposed two-step flow (with BrowserID/FxA assertions): 1. the client trades a BrowserID assertion for an Auth token and corresponding secret 2. the client uses the auth token to sign subsequent requests using Hawk Auth. Getting an Auth token:
seqdiag-8f2c0f2a4d333dd9d7424ad643ddc4670b0ee5f6.png
Calling the Service:
seqdiag-3610210bb1156abc60717e459e3fe6908a1a13c1.png
Detailed steps: • the client requests a token, giving its browser id assertion [1]:
GET/1.0/sync/request_token HTTP/1.1 Host: token.services.mozilla.com Authorization: Browser-ID
• The Login Server checks the BrowserID assertion [2] this step will be done locally without calling an external browserid server – but this could potentially happen (we can use PyBrowserID + use the BID.org certificate) The user’s email address is extracted, along with any Generation Number associated with the BrowserID cer- tificate. • The Login Server asks the Users DB for an existing record matching the users’ email address. If so, the allocated Node and previously-seen Generation Number are returned. • If the submitted Generation Number is smaller than the recorded one, the Login Server returns an error as the client’s BrowserID credentials are out of date. If the submitted Generation Number is larger than the recorded one, the Login Server updates the Users DB with the new value. • If the user is not allocated to a Node, the Login Server asks for a new one from the Node Assignment Server [4] • The Login Server creates a response with an Auth Token and corresponding Token Secret [5] and sends it back to the user. The Auth Token contains the user id and a timestamp, and is signed using the Signing Secret. The Token Secret is derived from the Master Secret and Auth Token using HKDF. It also adds the Node url in the response under api_endpoint [6]
2.6. Token Server 69 Mozilla Services Documentation, Release
HTTP/1.1 200OK Content-Type: application/json
{'id':
• The client saves the node location and hawkauth parameters to use in subsequent requests. [6] • For each subsequent request to the Service, the client calculates a special Authorization header using Hawk Auth [7] and sends the request to the allocated node location [8]:
POST/request HTTP/1.1 Host: some.node.services.mozilla.com Authorization: Hawk id=
• The node uses the Signing Secret to validate the Auth Token [9]. If invalid or expired then the node returns a 401 • The node calculates the Token Secret from its Master Secret and the Auth Token, and checks whether the signa- ture in the Authorization header is valid [10]. If it is invalid then the node returns a 401 • The node processes the request as defined by the Service [11]
History v1 - 2012-03-28
• Initial implementation
Resources
• Server: https://github.com/mozilla-services/tokenserver
Heka
Goal
Heka is a high-volume logging infrastructure designed to simplify data collection and analysis across multiple input sources and output formats. It is lightweight, but can be expanded through the addition of plugins written in Go or Lua. It can be used for: • Application performance metrics • Server load, memory consumption, and other machine metrics • Database, Cache server, and other daemon metrics • Log-file parsing and shipping
70 Chapter 2. Services Mozilla Services Documentation, Release
• Statsd-like time series data It is currently being used at Mozilla in the Marketplace and Sync infrastructures.
Resources
• Heka documentation: https://hekad.readthedocs.io • Heka binaries: https://github.com/mozilla-services/heka/releases • Heka source: https://github.com/mozilla-services/heka
Loop Server
Goal of the Service
Loop server allows firefox users to call each others via WebRTC. It is a rendezvous API built on top of an external service provider for NAT traversal and supplementary services.
Assumptions
• The Loop Server supports BrowserID authentication using FxA certificates and MSISDN certificates. It uses it on the /register endpoint to create an Hawk session. • All servers are time-synced • The Loop server keeps a white list of origins that can do Cross-Origin resource sharing.
Documentation content
Loop Server API v1.0
This document is based on the current status of the server. All the examples had been done with real calls. It doesn’t reflect any future implementation and tries to stick with the currently deployed version. This document describes the HTTP APIs and the Websockets APIs.
Contents
• Loop Server API v1.0 – HTTP APIs
* Overview * Versioning * Authentication · Derive hawk credentials from the hawk session token
* Configuration · GET /
2.8. Loop Server 71 Mozilla Services Documentation, Release
· GET /push-server-config
* Healthcheck · GET /__healthcheck__
* Registration · POST /registration · DELETE /registration
* Call URLs · GET /call-url · POST /call-url · PUT /call-url/{token} · DELETE /call-url/{token}
* Calls · GET /calls/{token} · POST /calls/{token} · POST /calls · GET /calls?version=
* Rooms · Basic Auth Authorization · POST /rooms · PATCH /rooms · PATCH /rooms/:token · DELETE /rooms/:token · POST /rooms/:token · Joining the room · Refreshing membership in a room · Leaving the room · Update Status · Shared Domain Logs · POST /events · GET /rooms/:token · GET /rooms · Participant information
* Channel information * Account and Session · DELETE /account
72 Chapter 2. Services Mozilla Services Documentation, Release
· DELETE /session
* Integration with Firefox Accounts using OAuth · POST /fxa-oauth/params · GET /fxa-oauth/token · POST /fxa-oauth/token
* Error Responses – Websockets APIs
* Call Setup States * Call Progress Protocol · Initial Connection (hello) · Call Progress State Change (progress) · Client Action (action)
* Termination Reasons * Timer Supervision · Server Timers · Supervisory Timer · Ringing Timer · Connection Timer · Client Timers · Response Timer · Media Setup Timer · Alerting Timer
HTTP APIs
Overview
Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types. Parameters for the GET requests are form encoded (?key=value&key2=value2)
To ease testing, you can use httpie in order to make requests. Examples of use with httpie are provided when possible. In order to authenticate with hawk, you’ll need to install the requests-hawk module
Versioning
The current API is versioned, using only a major version. All the endpoints for version 1 are prefixed by /v1/. In case you don’t specify the prefix, your requests will be redirected automatically with an http 307 status.
2.8. Loop Server 73 Mozilla Services Documentation, Release
Authentication
To deal with authentication, the Loop server uses Hawk sessions. When you register, you can do so with different authentications schemes, but you are always given an hawk session back, that you should use when requesting the endpoints which need authentication. When authenticating using the /register endpoint, you will be given an hawk session token in the Hawk-Session-Token header. You will need to derive it, as explained at Derive hawk credentials from the hawk session token.
Derive hawk credentials from the hawk session token
In order to get the hawk credentials to use on the client you will need to: 1. Do an HKDF derivation on the given session token. You’ll need to use the following parameters:
key_material= HKDF(hawk_session,"",'identity.mozilla.com/picl/v1/sessionToken', ˓→ 32*2)
2. The key material you’ll get out of the HKDF need to be separated into two parts, the first 32 hex caracters are the hawk id, and the next 32 ones are the hawk key. Credentials:
credentials={ 'id': keyMaterial[0:32] 'key': keyMaterial[32:64] 'algorithm':'sha256' }
If you are writting a client, you might find these resources useful: • With javascript: https://mxr.mozilla.org/mozilla-central/source/services/fxaccounts/FxAccountsClient.jsm#309 & https://github.com/mozilla/gecko-projects/blob/elm/browser/components/loop/content/shared/libs/token.js# L55-L77 • With python: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py#L99-L122
Configuration
GET /
Displays version information, for instance:
http GET localhost:5000/v1--verbose
GET /v1/ HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 247
74 Chapter 2. Services Mozilla Services Documentation, Release
Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 12:57:13 GMT ETag: W/"f7-762153207" Timestamp: 1405515433
{ "description": "The Mozilla Loop (WebRTC App) server", "endpoint": "http://localhost:5000", "fakeTokBox": false, "homepage": "https://github.com/mozilla-services/loop-server/", "name": "mozilla-loop-server", "version": "0.9.0" }
GET /push-server-config
Retrieves the configuration of the push server. Specifically, returns the websocket endpoint that should be used to reach simple push. The response should contain a pushServerURI parameter with this information.
http localhost:5000/push-server-config
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 57 Content-Type: application/json; charset=utf-8 Date: Tue, 19 Aug 2014 14:26:42 GMT ETag: W/"39-351294056" Timestamp: 1408458402
{ "pushServerURI": "wss://push.services.mozilla.com/" }
Server should acknowledge your request and answer with a status code of 200 OK.
Healthcheck
GET /__healthcheck__
• Returns 200 in case of success • Returns 503 with the backend error message in case backends are broken
http localhost:5000/__heartbeat__
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 32 Content-Type: application/json; charset=utf-8 Date: Fri, 07 Nov 2014 13:02:45 GMT ETag: W/"20-e938360a" Timestamp: 1415365365
2.8. Loop Server 75 Mozilla Services Documentation, Release
{ "provider": true, "storage": true }
Registration
POST /registration
Associates a Simple Push Endpoint (URL) with a user. Always return an hawk session token in the Hawk-Session-Token header. May require authentication You don’t need to be authenticated to register. In case you don’t register with a Firefox Accounts assertion or a valid hawk session, you’ll be given an hawk session token and be connected as an anonymous user. This hawk session token should be derived by the client and used for subsequent requests. You can currently authenticate by sending a valid Firefox Accounts assertion or a valid Hawk session. Body parameters: • simplePushURL, the simple push endpoint url as defined in https://wiki.mozilla.org/WebAPI/ SimplePush#Definitions Example (when not authenticated):
http POST localhost:5000/v1/registration--verbose \ simplePushURL=https://push.services.mozilla.com/update/MGlYke2SrEmYE8ceyu
POST /v1/registration HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Content-Length: 35 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "simplePushURL": "https://test" }
HTTP/1.1 200 OK Access-Control-Expose-Headers: Hawk-Session-Token Connection: keep-alive Content-Length: 4 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 12:58:56 GMT Hawk-Session-Token:
˓→c7ee533a75a4f3b8a2a44b0b417eec15295ad43ff2b402776078ec87abb31cd9 Timestamp: 1405515536
"ok"
Server should acknowledge your request and answer with a status code of 200 OK. Potential HTTP error responses include:
76 Chapter 2. Services Mozilla Services Documentation, Release
• 400 Bad Request: You forgot to pass the simple_push_url, or it’s not a valid URL. • 401 Unauthorized: The credentials you passed aren’t valid.
DELETE /registration
Requires authentication Unregister the given session’s SimplePushURLs. The server will not be able to notify the client for this session. Example:
http DELETE localhost:5000/v1/registration--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
DELETE /v1/registration HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 204 No Content Connection: keep-alive Date: Wed, 16 Jul 2014 13:03:39 GMT Server-Authorization:
Server should acknowledge your request and answer with a status code of 204 No Content. Potential HTTP error responses include: • 400 Bad Request: You forgot to pass the simplePushURL, or it’s not a valid URL. • 401 Unauthorized: The credentials you passed aren’t valid.
Call URLs
GET /call-url
Requires authentication List all user valid call-urls. Response from the server: The server should answer this with a 200 status code and a list of JSON objects with the following properties: • callerId The name of the person to whom the call-url has been issued ; • expires The date when the url will expire (the unix epoch, in seconds). • timestamp The date when the url has been created (the unix epoch, in seconds). Example:
2.8. Loop Server 77 Mozilla Services Documentation, Release
http GET localhost:5000/v1/call-url--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
GET /v1/call-url HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 186 Content-Type: application/json; charset=utf-8 Date: Thu, 13 Nov 2014 16:19:59 GMT Server-Authorization:
[ { "callerId": "Natim", "expires": 1416499576, "timestamp": 1415894776 } ]
Potential HTTP error responses include: • 401 Unauthorized: You need to authenticate to call this URL. • 503 Service Unavailable: Something is wrong on the server side.
POST /call-url
Requires authentication Generates a call url for the given callerId. This is an URL the caller can click on in order to call the caller. Body parameters: • callerId, the caller (the person you will give the link to) identifier. • expiresIn, the number of hours the call-url will be valid for. • issuer, The friendly name of the issuer of the token. Optional parameters: • subject, The subject of the conversation. Response from the server: The server should answer this with a 200 status code and a JSON object with the following properties: • callUrl The call url; • callToken The call token; • expiresAt The date when the url will expire (the unix epoch, in seconds).
78 Chapter 2. Services Mozilla Services Documentation, Release
Example:
http POST localhost:5000/v1/call-url--verbose \ callerId=Remy expiresIn=5 issuer=Alexis \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /v1/call-url HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "callerId": "Remy", "expiresIn": "5", "issuer": "Alexis", "subject": "MySubject" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 186 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 13:09:40 GMT Server-Authorization:
{ "callToken": "_nxD4V4FflQ", "callUrl": "http://localhost:3000/static/#call/_nxD4V4FflQ", "expiresAt": 1405534180 }
Potential HTTP error responses include: • 400 Bad Request: You forgot to pass the callerId, or it’s not valid; • 401 Unauthorized: You need to authenticate to call this URL.
PUT /call-url/{token}
Requires authentication Updates data associated with an already created call-url. Body parameters: • callerId, the caller (the person you will give the link to) identifier. The callerId is supposed to be a valid email address. • expiresIn, the number of hours the call-url will be valid for. • issuer, The friendly name of the issuer of the token. Optional parameters:
2.8. Loop Server 79 Mozilla Services Documentation, Release
• subject, The subject of the conversation. Response from the server: The server should answer this with a 200 status code and a JSON object with the following properties: • expiresAt The date when the url will expire (the unix epoch, in seconds). Example:
http PUT localhost:5000/v1/call-url/B65nvlGh8iM--verbose \ issuer=Adam--auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
PUT /v1/call-url/B65nvlGh8iM HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "issuer": "Adam", "subject": "MySubject2" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 29 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 14:16:54 GMT Server-Authorization:
{ "expiresAt": 1408112214 }
DELETE /call-url/{token}
Requires authentication Delete a previously created call url. You need to be the user who generated this link in order to delete it. Example:
http DELETE localhost:5000/v1/call-url/_nxD4V4FflQ--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
DELETE /v1/call-url/_nxD4V4FflQ HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
80 Chapter 2. Services Mozilla Services Documentation, Release
User-Agent: HTTPie/0.8.0
HTTP/1.1 204 No Content Connection: keep-alive Date: Wed, 16 Jul 2014 13:12:46 GMT Server-Authorization:
Potential HTTP error responses include: • 400 Bad Request: The token you passed is not valid or expired. • 404 Not Found: The token you passed doesn’t exist.
Calls
GET /calls/{token}
Returns information about the token. • token is the token returned by the POST on /call-url. Response from the server: The server should answer this with a 200 status code and a JSON object with the following properties: • calleeFriendlyName the friendly name the creator of the call-url gave. • urlCreationDate, the unix timestamp when the url was created. Optional: • subject, the subject of the conversation. Example:
http GET localhost:5000/v1/calls/3jKS_Els9IU--verbose
GET /v1/calls/3jKS_Els9IU HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 30 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 13:23:04 GMT ETag: W/"1e-2896316483" Timestamp: 1405516984
{ "calleeFriendlyName": "Alexis", "urlCreationDate": 1405517546, "subject": "MySubject" }
Potential HTTP error responses include:
2.8. Loop Server 81 Mozilla Services Documentation, Release
• 400 Bad Request: The token you passed is not valid or expired.
POST /calls/{token}
Creates a new incoming call for the given token. Gets tokens and session from the provider and does a simple push notification, then returns caller tokens. Body parameters: • callType, Specifies the type of media the remote party intends to send. Valid values are “audio” or “audio-video”. Optional parameters: • subject, the subject of the conversation • channel, the TokBox channel to use for the call. More information about the channel parameter. Server should answer with a status of 200 and the following information in its body (json encoded): • apiKey, the provider public api Key. • callId, an unique identifier for the call; • progressURL, the location to reach for websockets; • sessionId, the provider session identifier; • sessionToken, the provider session token (for the caller); • websocketToken, the token to use when authenticating to the websocket. Example:
http POST localhost:5000/v1/calls/QzBbvGmIZWU callType="audio-video"--
˓→verbose
POST /v1/calls/QzBbvGmIZWU HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Content-Length: 27 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "callType": "audio-video", "channel": "nightly", "subject": "MySubject" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 614 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 13:37:39 GMT Timestamp: 1405517859
{ "apiKey": "44669102", "callId": "35e7c3a511f424d3b1d6fba442b3a9a5",
82 Chapter 2. Services Mozilla Services Documentation, Release
"progressURL": "ws://localhost:5000/websocket", "sessionId": "1_MX40NDY2OTEwMn5-V2VkIEp1bCAxNiAwNjo", "sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaW", "websocketToken": "44ee04b9694ae121c03a1db685cfad6d" }
(note that return values have been truncated for readability purposes.) Potential HTTP error responses include: • 400 Bad Request: The token you passed is not valid. • 410 Gone: The token expired.
POST /calls
Requires authentication Similar to POST /calls/{token}, it creates a new incoming call to a known identity. Gets tokens and session from the provider and does a simple push notification, then returns caller tokens. Body parameters: • calleeId, array of strings containing the identities of the receiver(s) of the call. These identities should be one of the valid Loop identities (Firefox Accounts email or MSISDN) and can belong to none, an unique or multiple Loop users. It can also be an object with two properties: – phoneNumber The phone number on a local form – mcc The current SIM card Mobile Country Code In that case, the server will try to convert the phoneNumber as an MSISDN identity • callType, Specifies the type of media the remote party intends to send. Valid values are “audio” or “audio-video”. Optional parameters: • subject, the subject of the conversation • channel, the client channel to use for the call. More information about the channel parameter. Server should answer with a status of 200 and the following information in its body (json encoded): • apiKey, the provider public api Key. • callId, an unique identifier for the call; • progressURL, the location to reach for websockets; • sessionId, the provider session identifier; • sessionToken, the provider session token (for the caller); • websocketToken, the token to use when authenticating to the websocket. Example:
http POST localhost:5000/v1/calls--verbose \ calleeId=alexis callType="audio-video"\ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
2.8. Loop Server 83 Mozilla Services Documentation, Release
POST /v1/calls HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "callType": "audio-video" "calleeId": ["[email protected]", "+34123456789"], "channel": "nightly", "subject": "MySubject" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 614 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 13:37:39 GMT Server-Authorization:
{ "apiKey": "44669102", "callId": "35e7c3a511f424d3b1d6fba442b3a9a5", "progressURL": "ws://localhost:5000/websocket", "sessionId": "1_MX40NDY2OTEwMn5-V2VkIEp1bCAxNiAwNjo", "sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaW", "websocketToken": "44ee04b9694ae121c03a1db685cfad6d" }
(note that return values have been truncated for readability purposes.) Potential HTTP error responses include: • 400 Bad Request: You forgot to pass calleeId or is not valid. • 401 Unauthorized: You need to authenticate to call this URL.
GET /calls?version=
Requires authentication List incoming calls for the authenticated user since the given version. Querystring parameters: • version, the version simple push gave to the client when waking it up. Only calls that happened since this version will be returned. Server should answer with a status of 200 and a list of calls in its body. Each call has the following attributes: • apiKey, the provider public api Key. • callId, an unique identifier for the call. • callType, the call type (“audio” or “audio-video”).
84 Chapter 2. Services Mozilla Services Documentation, Release
• progressURL, the location to reach for websockets. • sessionId, the provider session identifier. • sessionToken, the provider session token (for the caller). • websocketToken, the token to use when authenticating to the websocket. Optional: • subject, the subject of the call In case of call initiated from an URL you will also have: • callToken, the call-url token used for this call. • callUrl, the call-url used for this call. • urlCreationDate, the unix timestamp when the used call-url was created.
GET /v1/calls?version=0 HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 1785 Content-Type: application/json; charset=utf-8 Date: Wed, 16 Jul 2014 14:10:38 GMT ETag: W/"6f9-2990115590" Server-Authorization:
{ "calls": [ { "apiKey": "44669102", "callId": "6744b8919d7d74e8c0b39590aa183565", "callToken": "QzBbvGmIZWU", "callUrl": "http://localhost:3000/static/#call/QzBbvGmIZWU", "call_url": "http://localhost:3000/static/#call/QzBbvGmIZWU", "callerId": "alexis", "progressURL": "ws://localhost:5000/websocket", "sessionId": "2_MX40NDY2OTEwMn5-
˓→V2VkIEp1bCAxNiAwNzoxMDoyMCBQRFQgMjAxNH4wLj", "sessionToken":
˓→"T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaWc9NzMyMGVmZjY1YWU0ZmFkZTY1NmU0", "urlCreationDate": 1405517546, "websocketToken": "a2fc1ee029169b62b08a4ba87c328d71", "subject": "MySubject" } ] }
Potential HTTP error responses include: • 400 Bad Request: The version you passed is not valid.
2.8. Loop Server 85 Mozilla Services Documentation, Release
Rooms
Some endpoints requires owner authentication, it is the account used to create the room on the POST /rooms. On these endpoints only the owner can perform the action on the room. Some endpoints requires participants authentification, it is either the Hawk Session used to join the room using the Hawk Authorization scheme or the sessionToken the user has got when joining anonymously using the Basic Auth Authorization scheme.
Basic Auth Authorization
In that case, just use the room participant sessionToken as a Basic Auth username with no password. http POST localhost:5000/rooms/:token –auth “_sessionToken_:” Authorization: Basic X3Nlc3Npb25Ub2tlbl86
POST /rooms
Requires owner authentication Creates a new room Request body parameters: • roomOwner, The room owner name. • maxSize, The maximum number of people the room can handle. At least one of the following two parameters must be supplied: • context, An encrypted room context string. • roomName, The name of the room, this is now obsolete, but remains to support older clients. Optional parameter: • expiresIn, the number of hours for which the room will exist. • channel, the client channel to use for the room. More information about the channel parameter. Response body parameters: • roomToken, The token used to identify the created room. • roomUrl, A URL that can be given to other users to allow them to join the room. • expiresAt, The date after which the room will no longer be valid (in seconds since the Unix epoch). Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example:
http POST localhost:5000/v1/rooms--verbose \ roomName="My Room" roomOwner="Natim" maxSize=5\ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
86 Chapter 2. Services Mozilla Services Documentation, Release
POST /rooms HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "maxSize": "5", "roomName": "My Room", "roomOwner": "Natim" }
HTTP/1.1 201 Created Connection: keep-alive Content-Length: 109 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:29:41 GMT Server: nginx/1.6.1 Server-Authorization:
{ "expiresAt": 1418221780, "roomToken": "pPVoaqiH89M", "roomUrl": "http://localhost:3000/static/#rooms/pPVoaqiH89M" }
PATCH /rooms
Requires owner authentication Remove given rooms Request body parameters: • deleteRoomTokens, a list of rooms to delete. Response body parameters: • responses, a mapping of room’s tokens and the request’s status for each. Potential HTTP error responses include: • 207 Multi-Status: When tokens are processed, each token having it’s own status. • 404 Not Found: If none of the given roomTokens where found for this user. • 400 Bad Requests: If no room tokens where provided. Example:
echo'{"deleteRoomTokens":["pPVoaqiH89M"]}'| http PATCH localhost:5000/v1/
˓→rooms-v \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
2.8. Loop Server 87 Mozilla Services Documentation, Release
PATCH /rooms HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "deleteRoomTokens": ["pPVoaqiH89M"] }
HTTP/1.1 207 Multi-Status Connection: keep-alive Content-Length: 40 Content-Type: application/json; charset=utf-8 Date: Tue, 30 Dec 2014 15:39:41 GMT Server: nginx/1.6.1 Server-Authorization:
{ "responses": { "pPVoaqiH89M": {"code": 200}, "_nxD4V4FflQ": {"code": 404, "errno": "105", "message": "Room not
˓→found."} }
PATCH /rooms/:token
Requires owner authentication Updates an existing room Optional request body parameters: • context, An encrypted room context string. • roomName, The name of the room. • roomOwner, The room owner name. • maxSize, The maximum number of people the room can handle. • expiresIn, the number of hours for which the room will exist. You only need set the body parameters you want to update. Response body parameters: • expiresAt, The date after which the room will no longer be valid (in seconds since the Unix epoch) Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example: http PATCH localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ roomName="My Room" roomOwner="Natim" maxSize=5\ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
88 Chapter 2. Services Mozilla Services Documentation, Release
PATCH /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "maxSize": "5", "roomName": "My Room", "roomOwner": "Natim" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 24 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:33:19 GMT Server: nginx/1.6.1 Server-Authorization:
{ "expiresAt": 1418221999 }
DELETE /rooms/:token
Requires owner authentication Deletes an existing room. Example:
http DELETE localhost:5000/v1/rooms/LURlwjMc8wI--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
DELETE /rooms/LURlwjMc8wI HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 204 No Content Connection: keep-alive Date: Mon, 10 Nov 2014 14:35:37 GMT Server: nginx/1.6.1 Server-Authorization:
2.8. Loop Server 89 Mozilla Services Documentation, Release
POST /rooms/:token
This endpoint handles three kinds of actions: • join, A new participant joins the room. • refresh, A participant notifies she is still in the room. • leave, A participant notifies she is leaving the room. • status, A participant update his status in the room. • logDomain, A participant send metrics about whitelisted shared domains.
Joining the room
Request body parameters: • action, Should be “join” in that case. • displayName, The participant friendly name for this room. • clientMaxSize, Maximum number of room participants the user’s client is capable of supporting. Response body parameters: • apiKey, The TokBox public api key. • sessionId, The TokBox session identifier (identifies the room). • sessionToken, The TokBox session token (identifies the room participant). • expires, The number of seconds within which the client must send another POST to this endpoint with the refresh action to remain a participant in this room. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example:
http POST localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ action=join displayName=Natim clientMaxSize=5\ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "action": "join", "clientMaxSize": "5", "displayName": "Natim" }
HTTP/1.1 200 OK
90 Chapter 2. Services Mozilla Services Documentation, Release
Connection: keep-alive Content-Length: 461 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:39:12 GMT Server: nginx/1.6.1 Server-Authorization:
{ "apiKey": "44669102", "expires": 300, "sessionId": "1_XM40NYDO2TEwMI5-
˓→MTQxNTYyOTc4MTIzOH5PaGxlZlNRTXdqVi9XRGUIel8jZWh0KZz-VH4", "sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEw...==" }
Refreshing membership in a room
Requires participant authentication Request body parameters: • action, Should be “refresh” in that case. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters • 410 Participation has expired: The referesh did not occur within the specified time period. Example:
http POST localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ action=refresh \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "action": "refresh" }
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 461 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:40:06 GMT Server: nginx/1.6.1 Server-Authorization:
2.8. Loop Server 91 Mozilla Services Documentation, Release
{ "expires": 300 }
Leaving the room
Request body parameters: • action, Should be “leave” in that case. The endpoint will return a 204 No Content response. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example:
http POST localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ action=leave \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "action": "leave" }
HTTP/1.1 204 No Content Connection: keep-alive Date: Mon, 10 Nov 2014 14:48:24 GMT Server: nginx/1.6.1 Server-Authorization:
Update Status
This endpoint is used to send some WebRTC metrics to be logged on server side. Request body parameters: • action, Should be “status” in that case. The endpoint will return a 204 No Content response. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example:
92 Chapter 2. Services Mozilla Services Documentation, Release
http POST localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ action=status event=Session.connectionCreated state=sendrecv \ connections=2 sendStreams=1 recvStreams=1\ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "action": "status", "event": "Session.connectionCreated", "state": "sendrecv", "connections": 2, "sendStreams": 1, "recvStreams": 1 }
HTTP/1.1 204 No Content Connection: keep-alive Date: Mon, 10 Nov 2014 14:48:24 GMT Server: nginx/1.6.1 Server-Authorization:
Shared Domain Logs
This endpoint is used to log domains that where shared with the Loop-client tab sharing feature. Request body parameters: • action, Should be “logDomain” in that case. The endpoint will return a 204 No Content response. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters Example:
echo'{ "action":"logDomain", "domains": [{"domain":"mozilla.org","count":1}, {"domain":"others":
˓→10}] }'| \ http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \ --auth-type=hawk --auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
POST /rooms/pPVoaqiH89M HTTP/1.1 Accept: application/json
2.8. Loop Server 93 Mozilla Services Documentation, Release
Accept-Encoding: gzip, deflate Authorization:
{ "action": "logDomain", "domains": [{ "domain": "mozilla.org", "count": 1 }, { "domain": "others", "count": 10 }] }
HTTP/1.1 204 No Content Connection: keep-alive Date: Mon, 10 Nov 2014 14:48:24 GMT Server: nginx/1.6.1 Server-Authorization:
POST /events
Requires authentication Event logging to the Google Analytics panel. • event, The event name • action, The action • label, The label The endpoint will return a 204 No Content response. Potential HTTP error responses include: • 400 Bad Request: Missing or invalid body parameters
http POST http://localhost:5000/v1/event-v \ event="Addon" action="start" label="Clicked Start Browsing"\ --auth-type=hawk--auth=
˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:'--json
POST /v1/event HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
{ "action": "start", "event": "Addon",
94 Chapter 2. Services Mozilla Services Documentation, Release
"label": "Clicked Start Browsing." }
HTTP/1.1 204 No Content Connection: keep-alive Date: Mon, 29 Feb 2016 14:39:41 GMT Server: nginx/1.6.1 Server-Authorization: Hawk mac="jcDymnhIosT8NzEdd+JZ9mwv4ZXHKDyfK/1+04gm5bA=" Vary: Origin
GET /rooms/:token
Retrieves information about the room. Response body parameters: • roomToken, The token used to identify this room. • context, An encrypted room context string. • roomName, The name of the room. • roomUrl, A URL that can be given to other users to allow them to join the room. • roomOwner, The user-friendly display name indicating the name of the room’s owner. If a participant authentication is provided, additional information is returned: • maxSize, The maximum number of users allowed in the room at one time (as configured by the room owner). • clientMaxSize, The current maximum number of users allowed in the room, as constrained by the clients currently participating in the session. If no client has a supported size smaller than “maxSize”, then this will be equal to “maxSize”. Under no circumstances can “clientMaxSize” be larger than “maxSize”. • creationTime, The time (in seconds since the Unix epoch) at which the room was created. • expiresAt, The time (in seconds since the Unix epoch) at which the room goes away. • participants, An array containing a list of the current room participants. More information about the participant properties. • ctime, The time, in seconds since the Unix epoch, that any of the following happened to the room: – The room was created – The owner modified its attributes with “PATCH /rooms/{token}” – A user joined the room – A user left the room Example:
http GET localhost:5000/v1/rooms/pPVoaqiH89M--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
2.8. Loop Server 95 Mozilla Services Documentation, Release
GET /rooms/pPVoaqiH89M HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 284 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:52:20 GMT ETag: W/"11c-d426a3d5" Server: nginx/1.6.1 Server-Authorization:
{ "clientMaxSize": 5, "creationTime": 1415629780, "ctime": 1415631010, "expiresAt": 1418221999, "maxSize": 5, "participants": [ { "displayName": "Natim", "roomConnectionId": "0bc7fa46-3df0-4621-b904-afdd2390d9ef", "owner": true, "account": "[email protected]" } ], "roomName": "My Room", "roomOwner": "Natim", "roomUrl": "http://locahost:3000/#/rooms/pPVoaqiH89M" }
GET /rooms
Requires owner authentication Retrieves a list of rooms owned by the owner. The response is a list of objects with this information: • roomToken, The token used to identify this room. • roomName, The name of the room. • context, An encrypted room context string. • roomUrl, A URL that can be given to other users to allow them to join the room. • roomOwner, The user-friendly display name indicating the name of the room’s owner. • maxSize, The maximum number of users allowed in the room at one time (as configured by the room owner).
96 Chapter 2. Services Mozilla Services Documentation, Release
• clientMaxSize, The current maximum number of users allowed in the room, as constrained by the clients currently participating in the session. If no client has a supported size smaller than “maxSize”, then this will be equal to “maxSize”. Under no circumstances can “clientMaxSize” be larger than “maxSize”. • creationTime, The time (in seconds since the Unix epoch) at which the room was created. • expiresAt, The time (in seconds since the Unix epoch) at which the room goes away. • participants, An array containing a list of the current room participants. More information about the participant properties. • ctime, The time, in seconds since the Unix epoch, that any of the following happened to the room: – The room was created – The owner modified its attributes with “PATCH /rooms/{token}” – A user joined the room – A user left the room Example:
http GET localhost:5000/v1/rooms--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
GET /rooms/ HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Content-Length: 312 Content-Type: application/json; charset=utf-8 Date: Mon, 10 Nov 2014 14:50:12 GMT ETag: W/"138-9bb2c1c" Server: nginx/1.6.1 Server-Authorization:
[ { "clientMaxSize": 5, "creationTime": 1415629780, "ctime": 1415631010, "expiresAt": 1418221999, "maxSize": 5, "participants": [ { "displayName": "Natim", "roomConnectionId": "0bc7fa46-3df0-4621-b904-afdd2390d9ef", "owner": true, "account": "[email protected]" } ],
2.8. Loop Server 97 Mozilla Services Documentation, Release
"roomName": "My Room", "roomOwner": "Natim", "roomToken": "pPVoaqiH89M", "roomUrl": "http://localhost:3000/static/#rooms/pPVoaqiH89M" } ]
Participant information
When retrieving the room information you get a list of participants. It is a list of objects with these properties: • displayName, The user-friendly name that should be displayed for this participant. • account, If the user is logged in, this is the FxA account name or MSISDN that was used to authen- ticate the user for this session. • owner, if the user is also the owner of the room, this property will be true, it will be false otherwise. • roomConnectionId, An id, unique within the room for the lifetime of the room, used to identify a partcipant for the duration of one instance of joining the room. If the user departs and re-joins, this id will change.
Channel information
The client can send its channel in order to let the server use different API keys (of the underlying service provider) depending on it. Channel can be one of: • release • esr • beta • aurora • nightly • default • mobile – used for the Firefox OS Mobile client • standalone – used for the standalone / “link-clicker” client
Account and Session
DELETE /account
Requires authentication Deletes the current account and all data associated to it. Example: http DELETE localhost:5000/v1/account--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
98 Chapter 2. Services Mozilla Services Documentation, Release
DELETE /v1/account HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 204 No Content Connection: keep-alive Date: Wed, 16 Jul 2014 13:03:39 GMT Server-Authorization:
DELETE /session
Requires authentication Deletes the current session. This should be used to clear the hawk session of a Firefox Account user. You should not attempt to call this endpoint with a non-firefox account session, since it would mean as a client you could not attach a session anymore. In case you want to destroy a non-FxA session, please use the DELETE /account endpoint. Example:
http DELETE localhost:5000/v1/session--verbose \ --auth-type=hawk--auth=
˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'
DELETE /v1/session HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 204 No Content Connection: keep-alive Date: Wed, 16 Jul 2014 13:03:39 GMT Server-Authorization:
Potential HTTP error responses include: • 403 Forbidden: If you remove this session you will loose access to your loop-server data because you will not be able to link them to a new session. Use DELETE /account instead.
Integration with Firefox Accounts using OAuth
A few endpoints are available for integration with Firefox Accounts. This is the prefered way to login with your Firefox Accounts for loop. For more information on how to integrate with Firefox Accounts, have a look at the Firefox Accounts documentation on MDN
2.8. Loop Server 99 Mozilla Services Documentation, Release
POST /fxa-oauth/params
Requires authentication Provide the client with the parameters needed for the OAuth dance. • client_id, the client id used by the server; • content_uri, URI of the content server (to get account information); • oauth_uri, URI of the OAuth server; • redirect_uri, URI where the client should redirect once authenticated; • scope, The scope of the token returned; • state, A nonce used to check that the session matches.
http POST http://localhost:5000/v1/fxa-oauth/params--verbose \ --auth-type=hawk--auth=
˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:'--json
POST /v1/fxa-oauth/params HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Server-Authorization:
{ "client_id": "263ceaa5546dce83", "content_uri": "https://accounts.firefox.com", "oauth_uri": "https://oauth.accounts.firefox.com/v1", "redirect_uri": "urn:ietf:wg:oauth:2.0:fx:webchannel", "scope": "profile", "state":
˓→"b56b3753c15efdcae80ea208134ecd6ae97f27027ce9bb11f7c333be6ea7029c" }
GET /fxa-oauth/token
Requires authentication Returns the current status of the hawk session (e.g. if it’s authenticated or not):
http GET http://localhost:5000/v1/fxa-oauth/token--verbose \ --auth-type=hawk--auth=
˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:'--json
If the current session is authenticated using OAuth, it returns it in the access_token attribute.
100 Chapter 2. Services Mozilla Services Documentation, Release
GET /v1/fxa-oauth/token HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization:
HTTP/1.1 200 OK Connection: keep-alive Content-Type: application/json; charset=utf-8 Server-Authorization:
POST /fxa-oauth/token
Requires authentication Trades an OAuth code with an oauth bearer token:
http POST http://localhost:5000/v1/fxa-oauth/token--verbose \ state=b56b3753c15efdcae80ea208134ecd6ae97f27027ce9bb11f7c333be6ea7029c \ code=12345\ --auth-type=hawk--auth=
˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:'--json
Checks the validity of the given code and state and exchange it with a bearer token with the OAuth servers. The token is returned in the access_token attribute. A few additional parameters are returned: • scope the scope of the token; • token_type the type of the token returned (here, it will be “bearer”).
Error Responses
All errors are also returned, wherever possible, as json responses with a code, errno and error message. Error status codes and codes and their corresponding outputs are: • 404 : unknown URL, or unsupported application. • 400 : malformed request. Possible causes include a missing option, bad values or malformed json. • 401 : you need to be authenticated • 403 [you are authenticated but don’t have access to the resource you are] requesting. • 405 : unsupported method • 406 : unacceptable - the client asked for an Accept we don’t support • 503 : service unavailable (provider or database backends may be down) Also the associated errno can be one of: • 105 INVALID_TOKEN: This come with a 404 on a wrong call-url token; • 106 BADJSON: This come with a 406 if the sent JSON is not parsable; • 107 INVALID_PARAMETERS: This come with a 400 and describe invalid parameters with a reason;
2.8. Loop Server 101 Mozilla Services Documentation, Release
• 108 MISSING_PARAMETERS: This come with a 400 and list all missing parameters; • 110 INVALID_AUTH_TOKEN: This come with a 401 and define a problem during Auth; • 111 EXPIRED: This come with a 410 and define a EXPIRE ressource; • 113 REQUEST_TOO_LARGE: This come with a 400 and define a too large request; • 114 INVALID_OAUTH_STATE: This come with a 400 and tells the oauth state is invalid; • 122 USER_UNAVAILABLE: This come with a 400 and tell the user could not be found in the database; • 201 BACKEND: This come with a 503 when a third party is not available at the moment.
Websockets APIs
During the setup phase of a call, the websocket protocol is used to let clients broadcast their state to other clients and to listen to changes. The client will establish a WebSockets connection to the resource indicated in the “progressURL” when it receives it. The client never closes this connection; that is the responsibility of the server. The times at which the server closes the connection are detailed below. If the server sees the client close the connection, it assumes that the client has failed, and informs the other party of such call failure. For forward compatibility purposes: • Unknown fields in messages are ignored • Unknown message types received by the client (indicating an earlier release) result in the client sending an “error” message ({“messageType”: “error”, “reason”: “unknown message”}). The call setup should continue. • Unknown message types received by the server result in the server sending an “error” message (as above); however, since this situation can only arise due to a misimplemented client or an out-of-date server, it results in call setup failure. The server closes the connection.
Call Setup States
Call setup goes through the following states:
102 Chapter 2. Services Mozilla Services Documentation, Release
Call Progress Protocol
Initial Connection (hello)
Upon connecting to the server, the client sends an immediate “hello” message, which serves two purposes: it identifies the call that the progress channel corresponds to (using the “callId”), as well as authenticating the connecting user, so that they can be verified to be authorized to view/impact the call setup state. Note that the callId with which this connection is to be associated is encoded as a component of the WSS URL. UA -> Server:
2.8. Loop Server 103 Mozilla Services Documentation, Release
{ "messageType":"hello", "auth":"''
• auth: Information to authenticate the user, so that they can be verified to be authorized to access call setup information. This is the websocketToken returned by a POST to /calls/{token}, POST /calls and GET /calls. If the hello is valid (the callId is known, the auth information is valid, and the authenticated user is a party to the call), then the server responds with a “hello.” This “hello” includes the current call setup state. Server -> UA:
{ "messageType":"hello", "state":"alerting" // may contain"reason" field for certain states. }
• state: See states in “progress”, below. If the hello is invalid for any reason, then the server sends an “error” message, as follows. It then closes the connection. Server -> UA:
{ "messageType":"error", "reason":"unknown callId" }
reason: The reason the hello was rejected: • unknown callId • invalid authentication - The auth information was not valid • unauthorized - The auth information was valid, but did not match the indicated callId
Call Progress State Change (progress)
The server informs users of the current state of call setup. The state sent to both parties ‘’is always the same state’‘. So, for example, when a user rejects a call, he will receive a “progress” message with a state of “terminated” and a reason of “rejected.” Server -> UA:
{ "messageType":"progress", "state":"alerting" // may contain optional"reason" field for certain events. }
Defined states are: • init: The call is starting, and the remote party is not yet being alerted. • alerting: The called party is being alerted (triggered by remote party sending a “hello” message).
104 Chapter 2. Services Mozilla Services Documentation, Release
• terminated: The call is no longer being set up. After sending a “terminated” message, the server closes the WebSockets connection. This message will include a “reason” field with one of the reason values described below. • connecting: The called party has indicated that he has answered the call, but the media is not yet confirmed • half-connected: One of the two parties has indicated successful media set up, but the other has not yet. • connected: Both endpoints have reported successfully establishing media. After sending a “connected” message, the server closes the WebSockets connection.
Client Action (action)
During call setup, clients send progress information about their own state so that it can be reflected in the call state. UA -> Server:
{ "messageType":"action", "event":"accept" // May contain"reason" field for certain events }
Defined event types are: • accept: Only sent by called party. The user has answered this call. This is sent before the called party attempts to set up the media. • media-up: Sent by both parties. Communications have been successfully established. • terminate: Sent by both parties. Ends attempt to set up call. Includes a “reason” field with one of values detailed below.
Termination Reasons
The following reasons appear in “action”/”terminate” and “progress” / “terminated” messages. The “” columns indi- cate whether the indicated element is permitted to generate the reason. When generated a “terminated” message as the result of receiving a “terminate” action from either client, the server will copy the reason code from the “terminate” action message into all resulting “terminated” progress messages, ‘’even if it does not recognize the reason code’‘. To provide for forwards compatibility, clients must be prepared to process “terminated” progress messages with un- known reason codes. The reaction to this situation should be the display of a generic “call setup failed” message. If the server receives an action of “terminate” with a reason it does not recognize, it copies that reason into the resulting “terminated” message. Reason CallerCalleeServerNote reject The called user has declined the call. busy The user is logged in, but cannot answer the call due to some current state (e.g., DND, in another call). timeout The call setup has timed out (The called party’s client has exceeded the amount of time it is willing to alert the user, or one of the server’s timers expired) cancel The calling party has cancelled a pending call. media-fail The called user has declined the call. closed The other user’s WSS connection closed unexpectedly. answered- This call has been answered on another device. elsewhere
2.8. Loop Server 105 Mozilla Services Documentation, Release
Timer Supervision
Server Timers
The server uses three timers to ensure that the call created by a setup attempt is cleaned up in a timely fashion.
Supervisory Timer
After responding to a `POST /call/{token}` or `POST /call/user` message, the server starts a supervi- sory timer of 10 seconds. • If the calling user does not connect and send a “hello” in this time period, the server considers the call to be failed. The called user, if connected, will receive a “progress”/”terminated” message with a reason of “timeout”. • If the called user does not connect and send a “hello” in this time period, the server considers the call to be failed. The calling user, if connected, will receive a “progress”/”terminated” message with a reason of “timeout”.
Ringing Timer
Upon receiving a “hello” from the called user, the server starts a ringing timer of 30 seconds. If the called user does not send an “accept” message in this time period, then both parties will receive a “progress”/”terminated” message with a reason of “timeout”.
Connection Timer
Upon receiving an “accept” from the called user, the server starts a connection timer of 10 seconds. If the call setup state does not reach “connected” in this time period, then both parties will receive a “progress”/”terminated” message with a reason of “timeout”.
Client Timers
Response Timer
Every client message triggers a response from the server: “hello” results in “hello” or “error”; and “action” will always cause a corresponding “progress” message to be sent. When the client sends a message, it sets a timer for 5 seconds. If the server does not respond in that time period, it disconnects from the server and considers the call failed.
Media Setup Timer
After sending a “media-up” action, the client sets a timer for 10 seconds. If the server does not indicate that the call setup has entered the “connected” state before the timer expires, the client disconnects from the server and considers the call failed.
Alerting Timer
We may wish to let users configure the maximum amount of time the call is allowed to ring (up to 30 seconds) before it considers it unanswered. This timer would start as soon as user alerting begins. If it expires before the call is set up, then the called party sends a “action”/”disconnect” message with a reason of “timeout.”
106 Chapter 2. Services Mozilla Services Documentation, Release
Administration Tools
Around loop-server code we have created some little tools that helps administrate the server in production and trou- bleshooting.
Redis Tools
This are tools built to administrate the Redis storage.
HawkSession expiration TTL
If you want know in how many seconds will an Hawk Session expire, you can use the test_ttl.js tool:
NODE_ENV=production node test_ttl.js
This will use the server environment configuration to read the expiration in the REDIS database. The output looks like:
NODE_ENV=loadtest node ttl_hawk.js
˓→634ecc3dc394170edbd8b2b6d3c4c4526a354b5eda8f9ed3abaeb3a89a0f83a8 redis-cli TTL hawk.4d8535d8dfbde737e611828a690d0881d8cfd2e3eddd0dd6cb6150990bd39b5b expire in 2591943 seconds
Number of keys of each types
If you want to have general information about the memory usage and know how many of each keys are used, you can use:
$ ./redis_usage.sh localhost [...]
# Memory used_memory:592032 used_memory_human:578.16K used_memory_rss:6832128 used_memory_peak:3169928 used_memory_peak_human:3.02M used_memory_lua:33792 mem_fragmentation_ratio:11.54 mem_allocator:jemalloc-3.4.1
[...]
Keys for spurl.* ======spurl.89272fca80167430382fae1a92e4e561e186db71ba0e37178a5f4cb8ce81fa6c.
˓→e7f1f6adf79ed64b6b48b64cf63d75b81ec66f9ac615884a5736b209266048c7 Total of keys: 5
[...]
2.8. Loop Server 107 Mozilla Services Documentation, Release
Number of callUrls per user
This script gives you the average number of callUrls per users:
$ pip install redis hiredis $ python callurls_per_user.py average is 6.5
Statsd
Loop-Server have got a number of Statsd counters and timers that can help monitor what’s going on in near real-time. Name Type Description loop.activated-users counter New Hawk Session created for a user. loop.call-urls counter New call-url creation. loop.simplepush.call counter Calls made to a SimplePush URL. loop.simplepush.call. counter SimplePush calls success and failures. (success|failures) loop.simplepush.call.{reason} counter Calls made to a SimplePush URL for a given reason. loop.simplepush.call.{reason}. counter Calls success and failures on SP urls for a (success|failures) given reason. loop.aws.write timer AWS S3 roomContext write calls. loop.aws.read timer AWS S3 roomContext read calls. loop.aws.remove timer AWS S3 roomContext deletion calls. loop.filesystem.write timer Filesystem roomContext write calls. loop.filesystem.read timer Filesystem roomContext read calls. loop.filesystem.remove timer Filesystem roomContext deletion calls. loop.tokbox.createSession timer TokBox createSession calls. You can also find the user flow in the wiki, at https://wiki.mozilla.org/Loop/Architecture.
Resources
• Server: https://github.com/mozilla-services/loop-server
MSISDN Gateway
Goal of the Service
MSISDN Gateway server allows people to log-on BrowserID applications using a validated phone number. MSISDN Gateway server propose multiple ways to validate the phone number with regards to the country, operator and validation cost.
Assumptions
• The MSISDN supports multiple verification flows with regards to the MSISDN and MCC/MNC codes. • The client can discover availables verification methods using the /discover endpoint.
108 Chapter 2. Services Mozilla Services Documentation, Release
• The client /register to get an Hawk session that will be valid until a /unregister call. • The client can verify one and only one number per session. • The client can use its Hawk session to generate as many BrowserID certificates (valid for maximum a day) as needed until the /unregister call, there is not automatic expiration of the Hawk session. • A mobile number can be validated by multiple session (one per device or per app. i.e: One for Loop and one for the Marketplace.)
Documentation content
MSISDN Gateway API v1.0
This document is based on the current status of the server. All the examples had been done with real calls. It doesn’t reflect any future implementation and tries to stick with the currently deployed version. This document describes the HTTP API and the SMS API.
HTTP APIs
Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types. Parameters for the GET requests are form encoded (?key=value&key2=value2)
To ease testing, you can use httpie in order to make requests. Examples of use with httpie are provided when possible. In order to authenticate with hawk, you’ll need to install the requests-hawk module
Authentication
To deal with authentication, the MSISDN Gateway server uses Hawk sessions. When you register, you can do so with different authentications schemes, but you are always given an hawk session back, that you should use when requesting the endpoints which need authentication. When authenticating using the /register endpoint, you will be given an hawk session token called msisdnSessionToken in the body. You will need to derive as explained at Derive hawk credentials from the hawk session token.
APIs
GET /
Displays version information, for instance:
http GET localhost:5000--verbose
GET / HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
2.9. MSISDN Gateway 109 Mozilla Services Documentation, Release
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 219 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 13:52:16 GMT ETag: W/"db-3773650714" Timestamp: 1406555536
{ "description": "The Mozilla MSISDN Gateway", "endpoint": "http://localhost:5000", "homepage": "https://github.com/mozilla-services/msisdn-gateway/", "name": "mozilla-msisdn-gateway", "version": "0.5.0" }
POST /register
Creates a new msisndSessionToken associated with an Hawk session. This is the first step to start verifying a new number. Example: http POST localhost:5000/register--verbose
POST /register HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Content-Length:0 Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 94 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 13:55:20 GMT Timestamp: 1406555720
{ "msisdnSessionToken":
˓→"8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee" }
Server should acknowledge your request, return a msisdnSessionToken and answer with a status code of 200 OK. Potential HTTP error responses include: • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
110 Chapter 2. Services Mozilla Services Documentation, Release
POST /unregister
Requires authentication Unregister an Hawk session. To revoke a device or once we don’t want to use this validation number anymore, we can unregister the session token to prevent the user from continuing to generate BrowserID certificates with it. Example:
http POST localhost:5000/unregister--verbose \ --auth-type=hawk \ --auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:
˓→'
POST /unregister HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization: Hawk mac="Tpny...a+A=", hash="B0we...z8=", id="bc...2f", ts=
˓→"1406556506", nonce="E_GRLT" Content-Length:0 Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 204 No Content Access-Control-Allow-Credentials: true Connection: keep-alive Date: Mon, 28 Jul 2014 14:08:26 GMT Server-Authorization: Hawk mac="lTGx...PNM=", hash="B0we...Uz8="
Server should acknowledge your request and answer with a status code of 204 No Content. Potential HTTP error responses include: • 401 Unauthorized: The credentials you passed aren’t valid. (errno: 109 or 110) • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
POST /discover
Discover which validation methods are available for a given MSISDN number or a MCC/MNC network- code. Body parameters: • mcc, the Mobile Country Code. • mnc, the Mobile Network Code (optional). • msisdn, the Mobile Station ISDN Number that is the user phone number to validate in its interna- tional form i.e 33623456789 (optional). Example (with MCC only):
2.9. MSISDN Gateway 111 Mozilla Services Documentation, Release
http POST localhost:5000/discover mcc=208--verbose
POST /discover HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Content-Length: 14 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "mcc": "214" }
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 169 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 14:18:05 GMT Timestamp: 1406557085
{ "verificationDetails": { "sms/momt": { "moVerifier": "+34191600777", "mtSender": "Mozilla@" } }, "verificationMethods": [ "sms/momt" ] }
Example (with all parameters):
http POST localhost:5000/discover msisdn=+3412578946 mcc=208 mnc=07--verbose
POST /discover HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Content-Length: 51 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "mcc": "214", "mnc": "07", "msisdn": "3412578946" }
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 286 Content-Type: application/json; charset=utf-8
112 Chapter 2. Services Mozilla Services Documentation, Release
Date: Mon, 28 Jul 2014 14:20:07 GMT Timestamp: 1406557207
{ "verificationDetails": { "sms/momt": { "moVerifier": "+34191600777", "mtSender": "Mozilla@" }, "sms/mt": { "mtSender": "Mozilla@", "url": "http://localhost:5000/sms/mt/verify" } }, "verificationMethods": [ "sms/mt", "sms/momt" ] }
Response from the server: The server should answer this with a 200 OK status code and a JSON object with the following properties: • verificationMethods, a list of verification methods available for the given set of parameters, in order of preferred use • verificationDetails, an object whose keys are the elements of verificationMethods and whose values are the details of each method The methods listed in verificationMethods are sorted in the preferred order from the perspective of the server, i.e., the method listed first is the most preferred method. Potential HTTP error responses include: • 400 Bad Request: MCC, MNC or MSISDN missing (errno: 108) or invalids (errno: 107) • 406 Bad JSON: The body is not valid JSON. (errno: 106) • 411 Length Required: Content-Length header wasn’t provided. (errno: 112) • 413 Request Too Large: Request Too Large. (errno: 113) • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
POST /sms/mt/verify
Requires authentication Starts the SMS MT flow by sending a SMS to the MSISDN to register Body parameters: • msisdn, the Mobile Station ISDN Number that is the user phone number to validate in its interna- tional form i.e +33623456789. • mcc, the Mobile Country Code. • mnc, the Mobile Network Code (optional).
2.9. MSISDN Gateway 113 Mozilla Services Documentation, Release
• shortVerificationCode, a parameter to ask a human transcribable 6 digits code if set to true. In that case the server will also take care of the Accept-Language header to localize any text in the SMS (optional) Response from the server: The server should answer this with a 204 No Content status code. Example:
http POST localhost:5000/sms/mt/verify msisdn=+33123456789 mcc=208\ --verbose \ --auth-type=hawk \ --auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:
˓→'
POST /sms/mt/verify HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization: Hawk mac="THYl...=", hash="mw68...=", id="9ee4...1a81", ts=
˓→"1406557901", nonce="xoIdtg" Content-Length: 53 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "mcc": "208", "msisdn": "+33123456789" }
HTTP/1.1 204 No Content Access-Control-Allow-Credentials: true Connection: keep-alive Date: Mon, 28 Jul 2014 14:31:41 GMT Server-Authorization: Hawk mac="wx9m...=", hash="B0we...="
Potential HTTP error responses include: • 400 Bad Request: MCC, MNC or MSISDN missing (errno: 108) or invalids (errno: 107) • 406 Bad JSON: The body is not valid JSON. (errno: 106) • 411 Length Required: Content-Length header wasn’t provided. (errno: 112) • 413 Request Too Large: Request too large. (errno: 113) • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
POST /sms/verify_code
Requires authentication Validates the code received by SMS Body parameters: • code, the code received by SMS.
114 Chapter 2. Services Mozilla Services Documentation, Release
Response from the server: The server should answer this with a 200 OK status code and a JSON object with the following properties: • msisdn The Mobile phone number that has been validated during the session. Example:
http POST localhost:5000/sms/verify_code
˓→code=15d3b227b0e58f216ee49b8da41c05c8 \ --verbose \ --auth-type=hawk \ --auth='c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:
˓→'
POST /sms/verify_code HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization: Hawk mac="8/Rg...=", hash="oAXy...=", id="9ee4...1a81", ts=
˓→"1406558280", nonce="WBjO3I" Content-Length: 44 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "code": "15d3b227b0e58f216ee49b8da41c05c8" }
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 30 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 14:38:00 GMT Server-Authorization: Hawk mac="nFOD...=", hash="NVuB...=" Timestamp: 1406558280
{ "msisdn": "+33123456789" }
Potential HTTP error responses include: • 400 Bad Request: code missing (errno: 108) or invalid (errno: 105) • 406 Bad JSON: the body is not valid JSON. (errno: 106) • 410 Expired: the session MSISDN has expired. (errno: 111) • 411 Length Required: Content-Length header wasn’t provided. (errno: 112) • 413 Request Too Large: Request too large. (errno: 113) • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
2.9. MSISDN Gateway 115 Mozilla Services Documentation, Release
POST /certificate/sign
Requires authentication Generate a BrowserID certificate with the given public key for the validated number. Example:
http POST localhost:5000/certificate/sign \ duration=3600\ publicKey='{"algorithm":"DS","y":"e6...40","p":"d6...01","q":"b1...3b","g
˓→":"9a...ef"}'\ --verbose \ --auth-type=hawk \ --auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:
˓→'
POST /certificate/sign HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate Authorization: Hawk mac="6vKD...=", hash="PKZT...=", id="9e...81", ts=
˓→"1406558679", nonce="IFjRIR" Content-Length: 1702 Content-Type: application/json; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/0.8.0
{ "duration": "3600", "publicKey": "{\"algorithm\":\"DS\",\"y\":\"e6...40\",\"p\":\"d6...01\",\
˓→"q\":\"b1...3b\",\"g\":\"9a...ef\"}" }
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 2602 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 14:44:39 GMT Server-Authorization: Hawk mac="QMCs...=", hash="NVuB...=" Timestamp: 1406558679
{ "cert": "eyJh...1Rgg" }
Potential HTTP error responses include: • 400 Bad Request: duration or publicKey parameter missing (errno: 108) or invalid (errno: 107) • 406 Bad JSON: The body is not valid JSON. (errno: 106) • 411 Length Required: Content-Length header wasn’t provided. (errno: 112) • 413 Request Too Large: Request Too Large. (errno: 113) • 429 Too Many Requests: Client has sent too many requests (errno: 117) • 503 Service Unavailable: Service temporarily unavailable due to high load or misconfiguration of the storage backend. (errno: 201)
116 Chapter 2. Services Mozilla Services Documentation, Release
GET /.well-known/browserid
Returns information for the BrowserID verifier. Response from the server: The server should answer this with a 200 status code and a JSON object with the following properties: • public-key the server public-key used to validate the BrowserId certificate • authentication, the link to the authentication page • provisionning, the link to the provisionning page Example:
http GET localhost:5000/.well-known/browserid--verbose
GET /.well-known/browserid HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 1815 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 14:54:50 GMT ETag: W/"717-781509924" Timestamp: 1406559290
{ "authentication": "/.well-known/browserid/warning.html", "provisioning": "/.well-known/browserid/warning.html", "public-key": { "algorithm": "DS", "g": "9a...ef", "p": "d6...01", "q": "b1...3b", "y": "7e...b9" } }
GET /api-specs
An endpoint that gives back the videur configuration. Server should answer with a status of 200 and the API routes as a JSON object. Example:
http GET localhost:5000/api-specs--verbose
GET /api-specs HTTP/1.1 Accept: */*
2.9. MSISDN Gateway 117 Mozilla Services Documentation, Release
Accept-Encoding: gzip, deflate Host: localhost:5000 User-Agent: HTTPie/0.8.0
HTTP/1.1 200 OK Access-Control-Allow-Credentials: true Connection: keep-alive Content-Length: 1069 Content-Type: application/json; charset=utf-8 Date: Mon, 28 Jul 2014 14:57:54 GMT ETag: W/"42d-3425668954" Timestamp: 1406559474
{ "service": { "location": "http://localhost:5000", "resources": { "/": { "GET": {} }, "/.well-known/browserid": { "GET": {} }, "/.well-known/browserid/warning.html": { "GET": {} }, "/__heartbeat__": { "GET": {} }, "/certificate/sign": { "POST": { "max_body_size": "10k" } }, "/discover": { "POST": { "max_body_size": "10k" } }, "/register": { "POST": { "max_body_size": "10k" } }, "/sms/momt/": { "GET": {}, "POST": { "max_body_size": "10k" } }, "/sms/mt/verify": { "POST": { "max_body_size": "10k" } }, "/sms/verify_code": { "POST": {
118 Chapter 2. Services Mozilla Services Documentation, Release
"max_body_size": "10k" } }, "/unregister": { "POST": { "max_body_size": "10k" } } }, "version": "0.5.0", "videur_version": "0.1" } }
SMS /sms/momt/verify
The SMS message to a moVerifier number will start the MOMT flow. SMS should be of the form of: /sms/momt/verify
/sms/momt/verify
˓→9ee442e5c8575c4077db786a40f603a70ae8eee09f5b41e34c096410f6fc1a81
• path, future proof path, in order to be able to route SMS to the right endpoint. • hawkId, the hawk session id extracted from the msisdnSessionToken using HKDF. When the SMS Gateway receive the Inbound Message, it will make a call on the configured endpoint. For Nexmo — GET or POST /sms/momt/?provider=nexmo Querystring parameters: • msisdn, the phone number from which the message is coming • text, the content of the message • network-code, the MCC/MNC unique identifier For BeepSend — GET or POST /sms/momt/?provider=beepsend Querystring or Body parameters: • from, the phone number from which the message is coming • message, the content of the message • mcc, The Mobile Country Code (in GET) • mnc, The Mobile Network Code (in GET) • mccmnc, {“mcc”: “
Error Responses
All errors are also returned, wherever possible, as json responses with a code, errno and error message.
2.9. MSISDN Gateway 119 Mozilla Services Documentation, Release
Error status codes and codes and their corresponding outputs are: • 404 : unknown URL or unsupported application. • 400 : malformed request. Possible causes include a missing option, bad values or malformed json. • 401 : you need to be authenticated • 403 [you are authenticated but don’t have access to the resource you are] requesting. • 405 : unsupported method • 406 : unacceptable - the client asked for an Accept we don’t support • 503 : service unavailable (provider or database backends may be down) Also the associated errno can be one of: • 105 INVALID_CODE: This come with a 404 on a wrong validation code; • 106 BADJSON: This come with a 406 if the sent JSON is not parsable; • 107 INVALID_PARAMETERS: This come with a 400 and describe invalid parameters with a reason; • 108 MISSING_PARAMETERS: This come with a 400 and list all missing parameters; • 109 INVALID_REQUEST_SIG: This come with a 401 and define a problem with the Hawk hash; • 110 INVALID_AUTH_TOKEN: This come with a 401 and define a problem during Auth; • 111 EXPIRED: This come with a 410 and define a EXPIRE ressource; • 112 LENGTH_MISSING: This come with a 411 and defined a missing Content-Length header. • 113 REQUEST_TOO_LARGE: This come with a 400 and define a too large request; • 201 BACKEND: This come with a 503 when a third party is not available at the moment.
Resources
• Server: https://github.com/mozilla-services/msisdn-gateway • Test webapp: https://github.com/ferjm/msisdn-verifier-client
120 Chapter 2. Services CHAPTER 3
Server-side development guide
Python Server Development Guide
This documentation is aimed at developers that want to work with an existing Mozilla Services server project, or want to create a new one that follow the same conventions.
Warning: This document is currently under heavy development.
Contents of the documentation:
Overview of Services Python applications
Services Python applications are all WSGI applications based on the same stack of tools:
NGinx<= TCP=> Gunicorn<= WSGI=> SyncServerApp<=> WebOb
• NGinx : A high-speed HTTP Server/reverse proxy. • GUnicorn: A Python WSGI Server. • SyncServerApp: A base WSGI application for all Services apps. Located in the server-core repository. • WebOb: a Request and a Response object with a simple interface. Services provides two base libraries to build WSGI applications: • cef: implements a CEF logger for ArcSight. • server-core: provides helpers to build Services applications.
121 Mozilla Services Documentation, Release
cef
Most Services applications need to generate CEF logs. A CEF Log is a formatted log that can be used by ArcSight, a central application used by the Infrasec team to manage application security. The cef module provides a log_cef() function that can be used to emit CEF logs: log_cef(message, severity, environ, config, [username, [signature]], **kw) Creates a CEF record, and emits it in syslog or another file. Args: • message: message to log • severity: integer from 0 to 10 • environ: the WSGI environ object • config: configuration dict • username: user name, defaults to ‘none’ • signature: CEF signature code, defaults to ‘AuthFail’ • extra keywords: extra keys used in the CEF extension Example:
>>> from cef import log_cef >>> log_cef('SecurityAlert!',5, environ, config, ... msg='Someone has stolen my chocolate')
With environ and config provided by the web environment. Note that the CEF library is published at PyPI: http://pypi.python.org/pypi/cef See CEF for more info on this. server-core
The server-core library provides helpers to build Services applications. In server-core‘s philosophy, a WSGI application is a SyncServerApp instance which will contain a config attribute that’s a mapper containing all the configuration needed by the code. This configuration is loaded from a unique ini-like file. When a request comes in, Routes is used to dispatch it to a controller method. Controllers are simple classes whose methods receive the request and need to return a response.
Configuration files
The configuration files we use in Services applications are based on an extended version of Ini files. You can find a description of the file at https://wiki.mozilla.org/Services/Sync/Server/GlobalConfFile. server-core provides a simple reader:
>>> from services.config import Config >>> cfg= Config('/etc/keyexchange/keyexchange.conf') >>> cfg.sections() ['keyexchange', 'filtering', 'cef']
122 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
>>> cfg.items('keyexchange') [('max_gets', 10), ('root_redirect', 'https://services.mozilla.com'), ('use_memory', False), ('ttl', 300), ('cache_servers', ['127.0.0.1:11211']), ('cid_len', 4)]
>>> cfg.get('keyexchange','cid_len') 4
Note that SyncServerApp will automatically create a Config instance over a central configuration file when it’s used to create a web application. See Configuration file for more info on this.
SyncServerApp
The SyncServerApp is a base class that can be used to get a few automation and some useful helpers when you want to create an application for Services. It provides: • a central configuration file • a pluggable authentication backend with an LDAP and an SQL plugin provided. • an overridable authentication process, defaulting to Basic Authentication. • a basic URL dispatcher based on Routes. • an error handler that ensures backend errors are logged and a 500 error is raised. • a heartbeat page useful for monitoring the server • a debug page to display useful information on the server • a few middlewares integrated: a profiler, an error catcher and a console logger. To create an application using SyncServerApp, see Complete layout.
Setting up a development environment
This section makes the assumption that you are under Mac OS X or Linux. A section for Windows might be added later.
Prerequisites
Setting up a development environment for Services consists of installing those packages: 1. make (installed by default on most Linuxes) 2. the latest gcc 3. Python 2.6 (installed by default under Debuntu, python26 under CentOS) 4. Python 2.6 headers (python2.6-dev under Debuntu, python26-devel under CentOS) 5. python26-profiler under Ubuntu 6. Mercurial (mercurial in most distros).
3.1. Python Server Development Guide 123 Mozilla Services Documentation, Release
7. virtualenv 8. Distribute 9. Flake8 10. Paste 11. PasteDeploy 12. MoPyTools One simple way to install all tools from 7. to 12. in your environment is to run the Distribute bootstrap script, then install MoPyTools:
$ curl -O http://python-distribute.org/distribute_setup.py $ python distribute_setup.py $ easy_install MoPyTools
This will pull all other tools for you and install them. Although, each project provides a Makefile that bootstraps this step and installs tools 8. to 12. automatically. So if you prefer, it should suffice to have only virtualenv and distribute installed system-wide:
$ curl -O http://python-distribute.org/distribute_setup.py $ python distribute_setup.py $ easy_install virtualenv
Setting up the Project Environment
Once you have all the above tools installed, working on a project consists of creating an isolated Python environment using Virtualenv and develop in it. Each project provides a Makefile that bootstraps this step, so you should not have to do it manually. For example, to create an environment for the Sync project, you can run:
$ hg clone http://hg.mozilla.org/services/server-full $ cd server-full $ make build
The code is currently developed and tested using python2.6, and it most likely will not work with other versions of python. The project Makefile uses the default python interpreter found on your $PATH. If your system default python version is not 2.6, you will need to create a python2.6 virtualenv to run make build. For example:
$ virtualenv --python=python2.6 ~/venvs/py26 $ source ~/venvs/py26/bin/activate $ make build
Once the environment has been created, you can do a sanity check by running all tests:
$ make test
Configuring Flake8
Flake8 can be used from the command-line, by simply running it over one or several files:
124 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
$ flake8 syncreg/controllers/ syncreg/controllers/user.py:44: 'urlunparse' imported but unused syncreg/controllers/user.py:44: 'urlparse' imported but unused syncreg/controllers/user.py:49: 'Response' imported but unused syncreg/controllers/user.py:55: 'get_url' imported but unused syncreg/controllers/user.py:276: undefined name 'environ' syncreg/controllers/user.py:276: undefined name 'config' syncreg/controllers/user.py:180:8: E111 indentation is not a multiple of four syncreg/controllers/user.py:240:1: 'UserController.change_password' is too complex
˓→(10) syncreg/controllers/user.py:318:1: 'UserController.do_password_reset' is too complex
˓→(11)
A simpler way to use it without having to think about it, is to configure Mercurial to call it every time you commit a change. To use the Mercurial hook on any commit or qrefresh, change your .hgrc file like this:
[hooks] commit= python:flake8.run.hg_hook qrefresh= python:flake8.run.hg_hook
[flake8] strict=0
If the strict option is set to 1, any warning will block the commit. When strict is set to 0, warnings are just displayed in the standard output. Using a non-strict mode is good enough: it will show you the issues without blocking your commits, so you can decide what should be done. In some case, you might need to simply silent the warnings. You can do this with NOQA markers: • all modules that starts with a # flake8: noqa comment line are skipped. • all lines that ends with a “# NOQA” comment are skipped as well.
Code layout
There are two code layouts: • the minimal layout: simple Python application. • the complete layout: Web application.
Minimal layout
The minimal layout is an empty application that includes all the boiler-plate code all our applications should have. A minimal Services’ project usually contains: • Makefile and build.py: used to build the environment, run tests, etc. • RPM spec: used to build the project’s RPM. • a directory for the Python code. • README: brief intro about the project • setup.py: defines Distutils’ options.
3.1. Python Server Development Guide 125 Mozilla Services Documentation, Release
• MANIFEST.in: defines extra options for Distutils.
Paster Template
You can create a new application layout by using the services_base Paster template provided by MoPyTools, which will ask you a few questions:
$ paster create -t services_base MyApp Selected and implied templates: MoPyTools#services_base A Mozilla Services application
Variables: egg: MyApp package: myapp project: MyApp Enter version (Version (like 0.1)) ['']: 0.1 Enter description (One-line description of the project) ['']: A cool app that does it Enter author (Author name) ['']: Tarek Enter author_email (Author email) ['']: [email protected] Enter url (URL of homepage (or Repository root)) ['']: http://hg.mozilla.org/services/
˓→myapp Creating template services_base ... Generating Application... ..
Once the application is generated, a default layout is created:
$ find MyApp/ MyApp/ MyApp/Makefile MyApp/setup.py MyApp/README.txt MyApp/pylintrc MyApp/myapp MyApp/myapp/tests MyApp/myapp/tests/test_sample.py MyApp/myapp/tests/__init__.py MyApp/myapp/__init__.py MyApp/MyApp.spec MyApp/build.py
You can build and test that the project is ready, by going in the directory and running:
$ make build test
Makefile
Every project should have a Makefile with these targets: • build: used to build the project in-place and all its dependencies. • test: used to run all tests and produce tests coverage and lint reports. • build_rpms: used to build all RPMs for the project.
126 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
The build target can take optional arguments to define for the project or any of its dependency that leaves in our repositories, a Mercurial tag. For example, to build the KeyExchange project with the “rpm-0.1-10” tag, and its ServerCore dependency with the tag “rpm-0.1-15”, one may call:
$ make build SERVER_KEYEXCHANGE=rpm-0.1.10 SERVER_CORE=rpm-0.1-15
The option name is the repository named in upper case, with the dashes (“-”) replaced by underlines (“_”). So “server- core” becomes “SERVER_CORE”. For more info on building and releasing, see Releasing an application. The test target runs the Nose test runner, and can be used to work on the code. It’s also used by Jenkins to continuously test your project. See 5. Build the app and all RPMS for more info on the building process.
RPM Spec file
The spec file that gets generated is used by “make build_rpm” to generate a RPM for your application. It contains all the required dependencies for a stock Services application, but will require that you add any new dependency your code needs. setup.py and MANIFEST.in
XXX
Complete layout
The complete layout contains all the things the minimal layout has, plus everything needed to make it a Web applica- tion: • etc/ : all default config files. • package/wsgiapp.py: the web application itself. • package/controller.py: the web controller to start adding features. • package/run.py: the bootstrap file used by Gunicorn to run the app. • package/tests/functional/: a minimal functional test using WebTest. XXX All Projects repositories are located in http://hg.mozilla.org/services
Development cycle
The usual cycle to add a feature or fix a bug is: 1. Make sure there’s a bug in Bugzilla. 2. Create a functional or unit-test, or both. Then change the code until the tests pass. 3. Ask for a review in the bug then push your changes. 4. Get ready to revert or fix.
3.1. Python Server Development Guide 127 Mozilla Services Documentation, Release
Note that ramping up a new project is a bit specific since you can’t really follow a per-feature cycle until it has reached a certain state.
1. A reference in Bugzilla
Every planned change to the code base of a project should start by adding a bug in Bugzilla. This is the central place where all discussions related to the changes, code reviews will happen. Here’s an example: https://bugzilla.mozilla.org/show_bug.cgi?id=631233
2. Writing the test and the change
Ideally, you should start to write a test that demonstrates the bug or the new feature. See Writing tests for more info on how to write tests. For bugs, it’s fairly easy: you need to write a test that reproduces the exact same problem, then fix the code until the code passes. For new features, a test that demonstrates how it works needs to be written. Running tests is done with the test target:
$ make test
It’s important to run all tests to make sure your changes are not breaking the code base elsewhere. You won’t be able to try out all possible execution environments of course, and that’s the job of the Jenkins CI server. make test needs to include the tests from all the project dependencies. That’s the case when you start a fresh project with a template.
External dependencies
In Jenkins the tests are running in an environment built from scratch so the tests should not: 1. depend on any file that is not in the checkout 2. use the LDAP backend - or if it does, mock anything that is trying to call ldap The test can use MySQL but the database should be configured to use sqlite rather than MySQL, since there is no MySQL server. MemCached can be used and if you have a specific volatile backend, it can potentially be installed (Redis, etc.). In any case, make sure you still have a fallback for those so anyone that checkouts a project can run the tests without having to install a third-party server. Note that Mozilla production backends are specifically tested via the functional tests that call the dev and stage clusters.
3. Ask for a review
XXX
4. Revert your change
The next steps are taken care of by Jenkins, who launch a test cycle to make sure that your change has not broken anything under every environment your code is used in. If it happens, an email is sent at services-builds.
128 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
In that case you need to fix the problem immediately, and if you can’t do it immediately, you need to revert all your changes.
Writing tests
A Services application must be tested in various ways: • unit and functional tests that are run locally • functional tests that are run on the dev, stage and production cluster
Writing unit tests
XXX explain unittest
Writing functional tests
XXX will demo WebTest
Running functional tests on a real server
XXX
Configuring Jenkins
XXX
Configuring the application
The application is configured via two files: • the Paster ini-like file, located in etc/ • the Services configuration file. XXX more on file location
Paster ini file
All Services projects provide a built-in web server that may be used to run a local instance. For example in server-full, once the project is built, you can run it:
$ bin/paster serve development.ini
This will run a server on port 5000. Paster reads an ini-like file that defines among other things: • where the wsgi application is located • where the application configuration is located • the logging configuration
3.1. Python Server Development Guide 129 Mozilla Services Documentation, Release
• etc. The main sections we want to configure in this file are: • DEFAULT • server:main • app:main • logging
DEFAULT
The default section defines four optional values (all are set to False by default): 1. debug: if set to True, will activate the debug mode. 2. client_debug: if set to True, will return in the body of the response any traceback when a 500 occurs. 3. translogger: will display in the stdout all requests made on the server. 4. profile: will activate a profiler and generate cachegrind infos. Example:
[DEFAULT] debug= True translogger= False profile= False server:main
Defines the web server to use to run the app with Paster. See Paster documentation for more info. Example:
[server:main] use= egg:Paste #http host= 0.0.0.0 port= 5000 use_threadpool= True threadpool_workers= 60 app:main
Defines the server entry point. See Paster documentation for more info. configuration can point to a configuration file for the server. It uses a file: prefix. Example:
[app:main] use= egg:SyncServer configuration= file:%(here)s/etc/sync.conf
130 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
logging
Logging is done using the logging configuration. See Python’s logging documentation for more details. The Sync server uses the syncserver logger everywhere. In the following example, all Sync errors are logged in a specific file as long as DEFAULT:debug is activated. Other logs are in a separate file: [loggers] keys = root,syncserver
[handlers] keys = global,errors
[formatters] keys = generic
[logger_root] level = WARNING handlers = global
[logger_syncserver] qualname = syncserver level = INFO handlers = global, errors propagate = 0
[handler_global] class = handlers.RotatingFileHandler args = ('/var/log/sync.log',) level = DEBUG formatter = generic
[handler_errors] class = handlers.RotatingFileHandler args = ('/var/log/sync-error.log',) level = ERROR formatter = generic
[formatter_generic] format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s datefmt = %Y-%m-%d %H:%M:%S
Configuring the profiler
Activates the repoze.profile middleware. XXX
Configuration file
The server uses a global configuration file. The file location is configured in the Paster ini file (as the configuration setting in the [app:main] section), and it is loaded when the application starts. See app:main. The configuration file has one section for each module loaded by the application. The configuration data will be available on the application object as a dictionary-like object stored in the config attribute. The settings from the
3.1. Python Server Development Guide 131 Mozilla Services Documentation, Release
[global] section will be stored as the simple key name, while the settings from the other sections will be keyed as
[storage] bing= boom
[auth] snaf= foo
Would produce the following app.config: {'foo':'bar', 'baz':'bawlp', 'storage.bing':'boom', 'auth.snaf':'foo'}
Additionally, config.get_section will return a dictionary containing only the settings from the specified section, without the prefix. So continuing the example above... app.config.get_section(‘global’) would return: {'foo':'bar', 'baz':'bawlp'} app.config.get_section(‘storage’) would return: {'bing':'boom}
Multi-Config Sections
In addition to supporting standard “INI” file conventions, Server-Core based applications provide an ability to use namespaced section headers to allow for multiple, similar variations of a config section to be registered simultaneously. The storage service used by the sync server is an example of where this is useful. Each storage server instance is associated with a specific set of back end storage nodes, usually all of the nodes that are located in the same location as the storage server itself. The storage server loads a separate configuration for each storage node, all similar save for the database connection URL. Example: [storage] backend= memcached cache_servers= 127.0.0.1:11211 192.168.1.13:11211 sqluri= mysql://localhost/sync standard_collections= false use_quota= true quota_size= 5120 pool_size= 100 pool_recycle= 3600
[host:node0]
132 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
storage.sqluri= mysql://sync:sync @db1.example.com/sync
[host:node1] storage.sqluri= mysql://sync:sync @db1.example.com/sync
[host:node2] storage.sqluri= mysql://sync:sync @db2.example.com/sync
[host:node3] storage.sqluri= mysql://sync:sync @db2.example.com/sync
The generated config object would then have sections named storage, host:node0, host:node1, etc., as you might expect. In addition to being available as separate sections, however, these configurations can be merged with the sections they refer to to generate “override” configs. These are made available via the merge(*sections) method on the config object. So, app.config would produce:
{'storage.backend':'memcached', 'storage.cache_servers':['127.0.0.1:11211','192.168.1.13:11211'], 'storage.sqluri':'mysql://localhost/sync', 'storage.standard_collections': False, 'storage.use_quota': True, 'storage.quota_size': 5120, 'storage.pool_size': 100, 'storage.pool_recycle': 3600, 'host:node0.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node1.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node2.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node3.storage.sqluri':'mysql://sync:[email protected]/sync', } while app.config.merge(‘host:node0’) would give:
{'storage.backend':'memcached', 'storage.cache_servers':['127.0.0.1:11211','192.168.1.13:11211'], 'storage.standard_collections': False, 'storage.use_quota': True, 'storage.quota_size': 5120, 'storage.pool_size': 100, 'storage.pool_recycle': 3600, 'storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node0.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node1.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node2.storage.sqluri':'mysql://sync:[email protected]/sync', 'host:node3.storage.sqluri':'mysql://sync:[email protected]/sync', }
Global
Global settings for the applications, under the global section. Available options (o: optional, m: multi-line, d: default): • retry_after [o, default:1800]: Value in seconds of the Retry-After header sent back when a 503 occurs in the application.
3.1. Python Server Development Guide 133 Mozilla Services Documentation, Release
• heartbeat_page [o, default:__heartbeat__]: defines the path of the heartbeat page. The heartbeat page is used by an HTTP Monitor to check that the server is still running properly. It returns a 200 if everything works, and a 503 if there’s an issue. A typical issue is the inability for the application to reach a backend server, like MySQL or OpenLDAP. • debug_page [o, default:None]: defines the path of the debug page. The debug page displays environ informa- tion. Warning: This page may expose private data. Once activated, it is not password-protected. If you use it, make sure the web server (Apache, Nginx) protects it from anonymous access. This feature is disabled by default to avoid any security issue. • shared_secret [o, default: None]: defines a secret string that can be used by the client when creating users, to bypass the captcha challenge. • graceful_shutdown_interval [o, default: 1]: Number of seconds before the app starts to shutdown. New requests are still accepted but the heartbeat page starts to return 503. • hard_shutdown_interval [o, default: 1]: Number of seconds before the app is shut down. Any new call returns a 503, and pending requests have that time to finish up the work before the app dies. Notice that an event is triggered when the process ends, giving a chance for apps to cleanup things. Example:
[global] retry_after= 60 heartbeat_page= __another_heartbeat_url_ debug_page= __debug__
Storage
The storage section is storage. It contains everything needed by the storage server to read and write data. Available options (o: optional, m: multi-line, d: default): • backend: backend used for the storage. Takes the fully qualified name of the class. Existing backends : – syncstorage.storage.sql.SQLStorage – syncstorage.storage.memcached.MemcachedSQLStorage. • cache_servers [o, m]: list of memcached servers (host:port) • sqluri: uri for the DB. see RFC-1738 for the format. driver://username:password@host:port/database. Sup- ported drivers are: sqlite, postgres, oracle, mssql, mysql, firebird • standard_collections [o, default: true]: if set to true, the server will use hardcoded values for collections. • use_quota [o, default:false]: if set to false, users will not have any quota. • quota_size [o, default:none]: quota size in KB • pool_size [o, default:100]: define the size of the SQL connector pool. • pool_recycle [o, default:3600]: time in ms to recycle a SQL connection that was closed. Example:
134 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
[storage]
backend= memcached cache_servers= 127.0.0.1:11211 192.168.1.13:11211
sqluri= mysql://sync:sync @localhost/sync standard_collections= false use_quota= true quota_size= 5120 pool_size= 100 pool_recycle= 3600
Authentication
The authentication section is auth. It contains everything needed for authentication and registration. Available options (o: optional, m: multi-line, d: default): • backend: backend used for the storage. Existing backends : – services.auth.sql.SQLAuth – services.auth.ldap.LDAPAuth – services.auth.dummy.DummyAuth • ldapuri [o]: uri for the LDAP server when the ldap backend is used. • ldap_use_pool [o, default:False]: If True, a pool of connectors is used. • ldap_pool_size [o, default:10]: Size of the ldap pool when used. • use_tls [o, default:false]: If set to true, activates TLS when using LDAP. • bind_user [o, default:none]: user for common LDAP queries. • bind_password [o, default:none]: password for the bind user. • admin_user [o, default:none]: user with extended rights for write operations. • admin_password [o, default:none]: password for the admin user. • users_root [o, default:none]: root for all ldap users. If set to md5 will generate a specific location based on the md5 hash of the user name. • cache_servers [o, m]: list of memcached servers (host:port) • sqluri: uri for the DB. see RFC-1738 for the format. driver://username:password@host:port/database. Sup- ported drivers are: sqlite, postgres, oracle, mssql, mysql, firebird • pool_size [o, default:100]: define the size of the SQL connector pool. • pool_recycle [o, default:3600]: time in ms to recycle a SQL connection that was closed. Example:
[auth] backend= ldap ldapuri= ldap://localhost:390
3.1. Python Server Development Guide 135 Mozilla Services Documentation, Release
ldap_timeout=-1 ldap_use_pool= true ldap_pool_size= 100
use_tls= false
bind_user="cn=admin,dc=mozilla" bind_password= admin
admin_user="cn=admin,dc=mozilla" admin_password= admin
users_root="ou=users,dc=mozilla"
sqluri= mysql://sync:sync @localhost/sync pool_size= 100 pool_recycle= 3600
cache_servers= 127.0.0.1:11211
Captcha
The captcha section enables the re-captcha feature during user registration. Available options (o: optional, m: multi-line, d: default): • use: if set to false, all operations will be done without captcha. • public_key: public key for reCaptcha. • private_key: private key for reCaptcha. • use_ssl: if set to true, will use SSL when connecting to reCaptcha. Example:
[captcha] use= true public_key=6Le8OLwSAAAAAK-wkjNPBtHD4Iv50moNFANIalJL private_key=6Le8OLwSAAAAAEKoqfc-DmoF4HNswD7RNdGwxRij use_ssl= false
Warning: The keys provided in this example work, as they were generated to provide a realistic example. But do not use them in your applications. Instead, you should generate a new set of keys for you own domain. See: https://www.google.com/recaptcha/admin/create
SMTP
The smtp section configures the SMTP connection used by the application to send e-mails. Available options (o: optional, m: multi-line, d: default): • host [o, default:localhost]: SMTP host
136 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
• port [o, default:25]: SMTP port • username [o, default:none]: SMTP user • password [o, default:none]: SMTP password • sender [o]: E-mail used for the sender field. Example:
[smtp] host= localhost port= 25 sender= weave @mozilla.com
CEF
The cef section configures how CEF security alerts are emitted. Available options (o: optional, m: multi-line, d: default): • use: if set to true, CEF alerts are emitted. • file: location of the CEF log file. Can be a file path or syslog to use the syslog facility. • syslog.options [o, default:none]: comma-separated values for syslog. Authorized values are: PID, CONS, NDELAY, NOWAIT, PERROR • syslog.priority [o, default:INFO]: priority level. Authorized values: EMERG, ALERT, CRIT, ERR, WARN- ING, NOTICE, INFO, DEBUG. • syslog.facility [o, default:LOCAL4]: facility. Authorized values: KERN, USER, MAIL, DAEMON, AUTH, LPR, NEWS, UUCP, CRON and LOCAL0 to LOCAL7. • vendor: CEF-specific option. • version: CEF-specific option. • device_version: CEF-specific option. • product: CEF-specific option. Example:
[cef] use= true file= syslog
syslog.options= PID,CONS syslog.priority= DEBUG syslog.facility= USER
vendor= mozilla version=0 device_version= 1.3 product= weave
Coding guidelines
3.1. Python Server Development Guide 137 Mozilla Services Documentation, Release
Style
All the code must follow PEP 8. You can use Flake8 to check for compliance. Flake8 is installed by default to all server environments, via MoPyTools. Flake8 provides a Mercurial hook, to perform a code check on every commit or qrefresh. Here’s an example of Mercurial ~/.hgrc file:
[hooks] commit= python:flake8.run.hg_hook qrefresh= python:flake8.run.hg_hook
[flake8] strict=0
If strict option is set to 1, any warning will block the commit. When strict is set to 0, warnings are just displayed in the standard output. Read these: • Flake8 documentation: http://pypi.python.org/pypi/flake8 • PEP 8 - The official style guideline: http://www.python.org/dev/peps/pep-0008
Unicode vs Str
All internal strings in our server applications must use the unicode type unless stated otherwise. Exceptions are: • Rendered e-mails • LDAP or SQL specific values When the str type is used, the utf-8 encoding must be used:
>>> uni=u'Café' >>> uni u'Caf\xe9' >>> uni.encode('utf-8') 'Caf\xc3\xa9' >>> print uni.encode('utf-8') Café >>> print uni Café
More on Unicode : http://docs.python.org/howto/unicode.html
Benching & Profiling
“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil” – D. Knuth
138 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
Profiling
The base application provides a Profiler that knows how to generate profiling data that you’ll be able to visualize in KCacheGrind. To enable it you can toggle the profile flag to True in your paster file. See Configuring the Profiler. Restart your application using the built-in Paster server and start to use it or run a load test on it. Remember that once enabled, your application will be really slow because the profiling is done by wrapping all calls in the Python interpreter. You can display the profile information in real-time by visiting http://localhost:5000/__profile__
But the real work should be done in KCacheGrind. Once you have finished testing the application, stop it and you should find at the root of your application a cachegrind.out file you can open.
3.1. Python Server Development Guide 139 Mozilla Services Documentation, Release
Benching
XXX
Releasing an application
Preparing a release
This section is a summary of the release process. The rest of the chapter explains everything in detail. To cut a new release the process is: 1. start a release branch 2. pin the external dependencies versions 3. update RELEASE.txt 4. tag a release candidate 5. build a release and all rpms
140 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
6. fix your release ! 7. backport your changes
1. Start a release branch
The first thing to do is to start a branch to do the releasing work. Let’s say you want to release 1.2, create a 1.2-release branch: $ git checkout -b 1.2-release
Go back to the default branch, then change the release to 1.3.dev1, so any further change will be in the 1.3 train. The version is located in the .spec file under the version field. Extract of a spec file: %define version 1.3.dev1
$ git checkout master ... edit the spec file so version=1.3.dev1... $ git commit -m 'starting the 1.3 developement' $ git push origin master
Once everything is committed, go back to the release branch: $ git checkout 1.2-release
Change the .spec file version to 1.2 (it’s probably 1.2.devX right now)
2. Pin the dependencies versions
When you release an application, you must make sure that all the dependencies are pinned. If you don’t do this, you can’t be sure the application will be run in stage or production with the same versions that the ones you’ve tested. This can be done in the prod-reqs.txt and stage-reqs.txt files. They contain all externals dependencies the project should be using. They usually have the same versions unless stage needs something specific. Example: cef == 0.2 WebOb == 1.0.7 Paste == 1.7.5.1 PasteDeploy == 1.3.4 PasteScript == 1.7.3 Mako == 0.4.1 MarkupSafe == 0.12 Beaker == 1.5.4 python-memcached == 1.47 simplejson == 2.1.6 Routes == 1.12.3 SQLAlchemy == 0.6.6 MySQL-python == 1.2.3 WSGIProxy == 0.2.2 recaptcha-client == 1.0.6
After you’ve tried a release, you can decide to raise the version to the latest stable version.
3.1. Python Server Development Guide 141 Mozilla Services Documentation, Release
3. Update RELEASE.txt
The RELEASE.txt file is a high-level changelog that’s used by other teams to know what the release contains. Each release has a section with the date, containing three parts: • Impacts: which teams are impacted by the release (Ops, QA, Infrasec, etc) • Dependencies: list internal dependencies and their versions. • Relevant changes: lists relevant changes with Bugzilla numbers. Example:
1.2- 2011-02-28 ======
Impacts:
- Ops
Dependencies:
- None
Relevant changes:
- Bug 636294- Prevent the automatic creation of the tables - now using the standalone cef lib
4. Tag the release
Our tags are following this scheme: “rpm-MAJOR.MINOR.[MICRO]” where MAJOR.MINOR[.MICRO] is the ver- sion of the Python package. Examples:
$ git tag rpm-1.2 $ git tag rpm-1.2.1
Note: The rpm- prefix is a legacy prefix we’re keeping to avoid any conflict with the old PHP version tags.
5. Build the app and all RPMS
Building the app can now be done, by providing the tag value for your app, and if needed a tag value for internal dependencies. For example for account-portal (uses server-core), a call can look like this:
$ make build_rpms SERVER_CORE=rpm-2.0 ACCOUNT_PORTAL=rpm-1.2 RPM_CHANNEL=stage
The syntax for the options is: PROJECT_NAME=rpm-X.X. When used, will checkout the given project at the mentioned tag. The tag can be a release tag, or master.
142 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
PROJECT_NAME refers to the name of the repository, after it has been upper-cased and all the dashes (“-”) replaced by underscores (“_”). For example, server-core becomes SERVER_CORE.
6. Fix your release
Sorry but your 1.2 release is a brown bag! You need to fix the spec file and maybe a few Python bugs. We will use the MICRO version to do this. • Increment the release to 1.2.1 • Do your fixes • Tag 1.2.1 • Repeat and increment the MICRO version until the release works
7. Backport your changes
If you did a few micro releases, check if you need to backport them to the default branch.
More details
Naming convention
To avoid any conflict with another Python project – even if the project will not be released to PyPI, let’s use these conventions: • The project name should start with MozSvc • Ideally a project contains a single package with a mozsvc prefix as well
Note: MozSvc is pronounced Mozz-Vikk, which is an ancient Irish Gaelic word that literally means “Viking Mice”.
Versioning scheme
Final Releases
For final releases, projects are versioned using the MAJOR.MINOR scheme. Examples: • 1.0 • 1.1 • 2.1 The MINOR part is incremented in the day-to-day work and the MAJOR part is incremented on important updates. The definition of important is left to the judgment of the releaser.
3.1. Python Server Development Guide 143 Mozilla Services Documentation, Release
We don’t really have any strategy here, like incrementing MAJOR only on backward incompatible changes: all Python packages we use are part of a server application and the only public facing API is documented web services that have their own versioning scheme. That said, if a library is published at PyPI, it has supposedly reached a stable state, and incrementing the MAJOR version should occur on backward incompatible changes. When a release fails in stage or prod, we can use a MAJOR.MINOR.MICRO scheme to fix it.
Development Releases
The master should always have a version with a .devN suffix. That is, the next version to be released, with N being an integer. Examples: • 1.5.dev1 • 1.4.dev23
Full example
Here’s a full scenario of versioning usage: • 1.2 is in production, tagged as “rpm-1.2” • we want to push a 1.3 • we change the default branch version to 1.4.dev1 • we branch “1.3-release” • a 1.3 is tagged there as “rpm-1.3” • 1.3 is pushed on stage • it’s not working • devs fix and tag 1.3.1 in the branch • 1.3.1 is pushed on stage, it’s working • 1.3.1 is pushed in production • it breaks !!! • production is rolled back to 1.2 • devs fix the problems and tag 1.3.2 • 1.3.2 is pushed on stage, it’s working • 1.3.2 is pushed in production • it works, congrats. Now working on 1.4.dev1 in master
The Makefile
Releases are driven by the Makefile file contained in the project. It should contain these targets: • build: builds the project in-place
144 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
• tests: runs the tests. • build_rpms: build the RPM collection. The collection must include the project RPM but also all direct and indirect dependencies. • mock: builds the RPMs, install them in a chroot, then make sure the app can be imported in Python In more details: The build target does the following: 1. install a local virtualenv 2. install MoPyTools in it 3. set the project to a specific channel (prod, dev or stage) 4. build the application and pull internal and external dependencies The test target runs Nose against the project. The build_rpms target generates the RPM for the project and for all its internal and external dependencies, using pypi2rpm The mock target calls build_rpms then installs everything in a chroot using Mock, then runs an import. That ensures the spec file dependencies are error free, and the Python app main module is importable. Notice that this target is run only under Centos5. Here’s an extract of a typical Makefile:
APPNAME = server-key-exchange DEPS = server-core BUILDAPP = bin/buildapp BUILDRPMS = bin/buildrpms CHANNEL = dev RPM_CHANNEL = prod VIRTUALENV = virtualenv NOSE = bin/nosetests -s --with-xunit TESTS = keyexchange/tests INSTALL = bin/pip install
build: $(VIRTUALENV) --no-site-packages --distribute . $(INSTALL) MoPyTools $(BUILDAPP) $(PYPIOPTIONS) -c $(CHANNEL) $(DEPS)
test: $(NOSE) $(TESTS)
build_rpms: $(BUILDRPMS) -c $(RPM_CHANNEL) $(DEPS)
mock: build build_rpms mock init mock --install python26 python26-setuptools cd rpms; wget http://mrepo.mozilla.org/mrepo/5-x86_64/RPMS.mozilla-services/
˓→gunicorn-0.11.2-1moz.x86_64.rpm cd rpms; wget http://mrepo.mozilla.org/mrepo/5-x86_64/RPMS.mozilla/nginx-0.7.65-4.
˓→x86_64.rpm mock --install rpms/* mock --chroot "python2.6 -m keyexchange.run"
3.1. Python Server Development Guide 145 Mozilla Services Documentation, Release
Channels
We define three channels: • dev: development channel, most dependencies are unpinned, so the latest PyPI release is taken • prod: all dependencies should be pinned default one • stage: all dependencies should be pinned – might vary from production versions. This channel is most of the time the same as production but can be useful in case the staging environment needs to be different.
Requirement files
All dependencies are listed in requirement files. A requirement file is a text file with a list of dependencies. One per line. Each dependency can have a version information. The file follows Pip’s standard. See http://www.pip-installer. org/en/latest/requirement-format.html Example: cef WebOb == 1.0.7 Paste PasteDeploy PasteScript Mako MarkupSafe Beaker python-memcached simplejson Routes SQLAlchemy<= 0.6.99 MySQL-python WSGIProxy recaptcha-client
There should be three requirement files located at the root of the project, one for each channel: 1. dev-reqs.txt: requirements for the dev channel 2. stage-reqs.txt: requirements for the stage channel 3. prod-reqs.txt: requirements for the prod channel stage and prod files should have pinned versions, since those files will be used to build applications to be released in production. Example: cef == 0.2 WebOb == 1.0.7 Paste == 1.7.5.1 PasteDeploy == 1.3.4 PasteScript == 1.7.3 Mako == 0.4.1 MarkupSafe == 0.12 Beaker == 1.5.4 python-memcached == 1.47 simplejson == 2.1.6 Routes == 1.12.3
146 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
SQLAlchemy == 0.6.6 MySQL-python == 1.2.3 WSGIProxy == 0.2.2 recaptcha-client == 1.0.6
When a build or a build_rpms is invoked, it receives a channel option and picks the corresponding requirement files to decide which version to pick. Unpinned versions will make the build process pick the latest release at PyPI. (Even if it’s not stable!) For the build target the default value is dev and for the build_rpms option it’s prod. You can also force a specific channel for build with the CHANNEL variable:
$ make build CHANNEL=prod
And for build_rpms, RPM_CHANNEL:
$ make build RPM_CHANNEL=stage
When the channel option is provided, the Makefile will use the dependencies list from the CHANNEL-reqs.txt file.
Configuration files specification
All Services applications need to use the same configuration file format. This document specifies it.
Syntax
The configuration file is a ini-based file. (See http://en.wikipedia.org/wiki/INI_file for more details.) Variable names can be assigned values, and grouped into sections. A line that starts with “#” is commented out. Empty lines are also removed. Example:
[section1] # comment name= value name2="other value"
[section2] foo= bar
Ini readers in Python, PHP and other languages understand this syntax. Although, there are subtle differences in the way they interpret values and in particular if/how they convert them.
Values conversion
Here are a set of rules for converting values: • If value is quoted with ” chars, it’s a string. This notation is useful to include “=” characters in the value. In case the value contains a ” character, it must be escaped with a “” character. • When the value is composed of digits and optionally prefixed by “-”, it’s tentatively converted to an integer or a long depending on the language. If the number exceeds the range available in the language, it’s left as a string.
3.1. Python Server Development Guide 147 Mozilla Services Documentation, Release
• If the value is “true” or “false”, it’s converted to a boolean, or 0 and 1 when the language does not have a boolean type. • A value can be an environment variable : “${VAR}” is replaced by the value of VARif found in the environment. If the variable is not found, an error must be raised. • A value can contain multiple lines. When read, lines are converted into a sequence of values. Each new line for a multiple lines value must start with at least one space or tab character. Examples:
[section1] # comment a_flag = True a_number = 1 a_string = "other=value" another_string = other value a_list = one two three user = ${USERNAME}
Extending a file
An INI file can extend another file. For this, a “DEFAULT” section must contain an “extends” variable that can point to one or several INI files which will be merged into the current file by adding new sections and values. If the file pointed to in “extends” contains section/variable names that already exist in the original file, they will not override existing ones. file_one.ini:
[section1] name2="other value"
[section2] foo= baz bas= bar
file_two.ini:
[DEFAULT] extends= file_one.ini
[section2] foo= bar
Result:
[section1] name2="other value"
[section2] foo= bar bas= bar
To point to several files, the multi-line notation can be used:
148 Chapter 3. Server-side development guide Mozilla Services Documentation, Release
[DEFAULT] extends= file_one.ini file_two.ini
When several files are provided, they are processed sequentially. So if the first one has a value that is also present in the second, the second one will be ignored. This means that the configuration goes from the most specialized to the most common.
Implementations
There’s one implementation in the core package of the Python server, but it could be moved to a standalone distribution if another project wants to use it. http://bitbucket.org/tarek/sync-core/src/tip/services/config.py
Glossary
WSGI Web Server Gateway Interface. It is a specification for web servers and application servers to communicate with web applications (though it can also be used for more than that). It is a Python standard, described in detail in PEP 333. Basic Authentication The client authenticates by sending an Authorization header of the form: “Basic base64(username + ‘:’ + password) services-builds The Build mailing list at [email protected] receives all Jenkins failures. You can subscribe to it at: https://mail.mozilla.org/listinfo/services-builds services-dev The Dev mailing list at [email protected] is the place where we discuss the development of Mozilla Services applications. You can subscribe to it at: https://mail.mozilla.org/listinfo/services-dev Feedback is welcome at [email protected] or at services-dev.
Indices and tables
• genindex • search
3.1. Python Server Development Guide 149 Mozilla Services Documentation, Release
150 Chapter 3. Server-side development guide CHAPTER 4
Client Development
Firefox Health Report
Content on this page has moved to https://hg.mozilla.org/mozilla-central/file/default/services/healthreport/docs/.
Metrics Collection
Content on this page has moved to https://hg.mozilla.org/mozilla-central/file/default/services/docs/.
Sync Client Documentation
This section is intended to provide a comprehensive guide to how Firefox Sync clients interact with the server and ultimately with each other to provide the functionality of syncing browser data between clients. It is a somewhat technical document, but should require no in-depth knowledge. Links to more detailed API docs offer an opportunity to dig deeper.
Overview
Introduction
The purpose of Sync is to exchange browser data (bookmarks, history, open tabs, passwords, add-ons, and the like) between “clients” in a manner that respects a user’s security and privacy. Syncing is facilitated through the use of a server, where data is centrally stored. This allows for syncing to occur without pairwise interaction between network-connected clients.
151 Mozilla Services Documentation, Release
Client 1 Client 2
Sync Server
Sync is different from most storage-in-the-cloud services in that data is encrypted locally - that is it cannot be read by other parties - before it is sent to the cloud. While many services encrypt data as it is being transmitted, Sync keeps your data encrypted even after it has arrived at the server. This means that the Sync server operators can’t read your data - even if they wanted to. The only way your data can be read is if someone possesses your secret Sync Key (sometimes referred to as a Recovery Key). This can occur if your device is lost or hacked or if you reveal it to another party. The important fact to note is that the Sync Key is never made available to the Sync Server and without it, your encrypted data is statistically impossible to recover. That being said, the server operators do have access to some limited data. This includes logs of when you connected and the types, number, and rough size of items being synchronized. This type of information is “leaked” for practically every network-connected service, so it shouldn’t come as a surprise.
The Sync Server
The Sync server performs the vital role of storing data, tracking elementary metadata, and providing authenticated access. The Sync server is effectively a dumb shared whiteboard - a bit bucket if you will. It plays a very small role in the actual syncing process. And, it must be this way: since data is encrypted before being sent to the server, there is not much the server can do to help. The Sync server infrastructure exposes a secure HTTP interface for user management and node assignment as well as storage. The storage server is actually a generic service that isn’t Sync-specific. Sync just uses it with specific semantics for how and where to store data. These semantics are fully described at Sync Storage Formats. Per-user access to the Sync server is protected via authentication at the HTTP layer. This can be whatever the server operator desires. Since the bulk of Sync’s security model resides in client-side encryption (read below) and since a Sync server is typically made available behind transport-level encryption (like SSL/TLS), primitive forms of security such as HTTP Basic Authentication are adequate. In fact, Mozilla’s hosted Sync server that is used by default by Firefox has used HTTP basic auth.
Collections and records
The primary concept behind the Sync server’s storage part is that of the collection. Clients can store objects called records into collections. Sync clients take their data, convert it to records, then send them to the Sync server. Receiving data does the same, but in reverse.
152 Chapter 4. Client Development Mozilla Services Documentation, Release
Records contain basic public metadata, such as the time they were last modified. This allows clients to selectively retrieve only the records that have changed since the last sync operation. An important observation is that the server has no notion of a “sync” as understood by the client. From the server’s perspective, there is simply a series of HTTP requests arriving from various IP addresses performing storage operations on a stateful backing store. The client has a well-defined sequence of actions that take place within a notional session, which can succeed or fail as a whole; the server does not.
Sync Clients
A Sync Client is an entity that talks to servers providing Sync functionality. Sync clients come in many different flavors with different levels of support for different features. For example, some clients may be read-only. A specific client often targets specific versions of the storage service and Sync storage formats.
The Life of a Sync
This document essentially describes how to write a Sync client. Because the Sync server is essentially a dumb storage bucket, most of the complexity of Sync is the responsibility of the client. This is good for users’ data security. It is bad for people implementing Sync clients. This document will hopefully alleviate common issues and answer common questions. Strictly speaking, information in this document applies only to a specific version of the Sync server storage format. In practice, client behavior is similar across storage versions. And, since we wish for clients to support the latest/greatest versions of everything, this document will target that.
Initial Client Configuration
The process of performing a sync starts with configuring a fresh client. Before you can even think about performing a sync, the client needs to possess key pieces of information. These include: • The URL of the Sync server. • Credentials used to access the Sync server. Depending on the versions of the Sync server and global storage version, you may also need a Sync Key or similar private key which is used to access encrypted data on an existing account. Obtaining these pieces of information is highly dependent on the server instance you will be communicating with, the client in use, and whether you are creating a new account or joining an existing one.
How Mozilla and Firefox Does It
For reference, this section describes how Mozilla and Firefox handle initial client configuration. Inside Firefox there exists a UI to Set up Firefox Sync. The user chooses whether she is setting up a new account or whether she wants to connect to an existing account. For completely new accounts, the user is presented with a standard sign-up form. The user enters her email address and selects a password. Behind the scenes Firefox is talking to a user provisioning service and the account is created there and a Sync server is assigned (Mozilla exposes many different Sync server instances to the Internet and the client connects directly to just one of them). At this time, a new Sync Key encryption key is generated and stored in Firefox’s credential manager (possibly protected behind a master password).
4.3. Sync Client Documentation 153 Mozilla Services Documentation, Release
If the user selects an existing account, the user is presented 12 random characters. These are entered on another device and the two devices effectively pair and share the login credentials, Sync Key, and server info. This is done with J- PAKE, so the data is secure as it is transported between devices. Even the intermediary agent bridging the connection between the two devices can’t decrypt the data inside.
Performing a Sync
Settings and State Pre-check
To perform a sync, a client will first need to perform some basic checks: • Do we have all necessary credentials? - Storage server HTTP credentials - Sync Key • Are we online (do we have network connectivity) • Are we prohibited from syncing due to result from a previous sync? - The server may have issued a backoff telling us to slow down, etc If these are all satisfied, the client can move on to the next phase.
Inspect and Reconcile Client and Server State
The initial requests performed on the Sync server serve to inspect, verify, and reconcile high-level state between the client and server.
Fetch info/collections
The first request to the Sync server should be a GET on the info/collections URI. This is a utility API provided by the storage service that reveals which collections exist on the server and when they were last modified. If the client has synced before, it should issue a conditional HTTP request by adding an X-If-Modified-Since header to the request. If the server responds with a 304, it means that no modifications have been made since the last sync. If the client has no new data to upload (perhaps it was just checking to see if there was any new data it needed to download), it can stop the sync right now: there is nothing more for it to do! The info/collections request also serves as a means to verify that the local credentials can connect with the server. If the server issues a 401 or 404 response, the client should interpret this as credentials failure. The next steps is this case are highly dependent on how the Sync server is configured. If using some kind of cached credentials (such as a token), the client may want to automatically try to fetch new credentials and try again. Assuming you have a response from info/collections, you’ll need to process that response and possibly take action. If you received a 304 and have data to upload, you can potentially skip processing if you have all the required values cached locally.
154 Chapter 4. Client Development Mozilla Services Documentation, Release
Prepare HTTP Request
Have Synced Before?
Yes
Add X-If-Modified-Since Header No
Perform HTTP Request
Wait for Response
Check Response
304 401, 403
Have Outgoing Changes? Reauthenticate
No Yes
End Sync Next Step
Validate meta/global
The client needs to validate the meta/global record on every request. Upon successful completion of the info/collections request, the following outcomes are possible: 1. The meta collection does not exist. 2. The meta collection has been modified since the last sync. 3. The meta collection has not been modified since the last sync.
4.3. Sync Client Documentation 155 Mozilla Services Documentation, Release
If the meta collection does not exist, the global record inside of it cannot exist. This means no client has synced yet. If info/collections reveals any collection exists, the client should issue a request to delete all data from the server to ensure the server is in a fresh state. If there are no collections on the server, you don’t need to issue a delete. Before we talk about uploading a new meta/global record, let’s talk about processing existing ones. If the meta collection has not been modified since the last sync and we have all of the data from a previous fetch of the meta/global cached locally (scenario 3), the client doesn’t need to do anything. If the meta collection has been modified or if the client doesn’t have a cached copy of the metaglobal data, the client will need to fetch the meta/global record. Simply issue a GET request to the appropriate URI and decode the payload according to the rules for the storage version the client is using. If you can’t decode the payload, that’s bad and should never happen. But, it is possible, so you need to handle it. One solution is to delete all data from the server and upload a new record. However, data on the server could be from a newer client this one just can’t understand, so it shouldn’t do this lightly. The storage versions have been defined such that the decoding format of the meta/global are backwards compatible with prior versions. So, if there is an error decoding, there is almost certainly something wrong going on. From the decoded payload, the client should first inspect the storage version number. If the client supports this storage version, all is well. Carry on. If not, the client has a few choices to make. If the version is older than what the client supports, the client can upgrade the server’s data to the new version. These semantics are highly specific to the specific version change. If the version is newer than what the client supports, the client should likely interpret this as “there is a newer client out there - I’m too old and need to upgrade.” If clients see a new storage format, they should probably stop what they are doing. Under no circumstances should clients attempt to modify data belonging to a newer storage version. Instead, delete all data and perform a fresh start (if this is really what you want to do). This section is incomplete. There is more that needs to be described. The graph below is also incomplete.
Check info/collections
No 'meta' collection
Any Collections Exist?
Yes Have 'meta' collection
Delete all Server Collections Modified Since Last Sync?
Wait for Response No
Process Response TODO
401, 403 204 No Content
Start New Sync Fresh Start
156 Chapter 4. Client Development Mozilla Services Documentation, Release
Validate crypto/keys
Have Keys Cached?
Yes
crypto Collection Exists? No
Yes No
crypto Collection Modified? WUT? crypto Collection Exists?
Yes Yes
Fetch Keys
200 OK No
Keys Valid?
No Not OK
401, 403 Generate Keys
OK
Upload Keys
201 No Context 401, 403
Next Step Abort
Collections Pre-Sync
Once meta/global and the cryptographic keys are in a good state, it is time to start syncing the regular collections. The first thing the client does is record the last modified times from the info/collections record. The client will ask the server for records that changed between the last time it synced and the last modified time of the collection.
Clients Collection
The clients collection, while a regular collection, is special. Clients always cache all records in the clients collection. This collection is also used to send commands between clients. Some commands tell a client to do important things, like clear data. Because of this, commands need to be processed before other collections.
4.3. Sync Client Documentation 157 Mozilla Services Documentation, Release
OLD CONTENT
Don’t read below this. It is old and needs to be formalized.
Perform sync
//- update engine last modified timestamps from info/collections record //- sync clients engine //- clients engine always fetches all records //- process reset/wipe requests in 'firstSync' preference //- process any commands, including the'wipeClient' command //- infer enabled engines from meta/global //- sync engines //- only stop if 401 is encountered //- if meta/global has changed, reupload it
Syncing an engine
// TODO WRITEME
// - meta/global // - syncID // - engine storage format // - fetch incoming records - GET .../storage/
Sync Storage Formats
The way that Sync clients store data on a Storage Server is defined by sets of integer storage versions. Each storage version defines specific semantics for how clients are supposed to behave.
158 Chapter 4. Client Development Mozilla Services Documentation, Release
Global Storage Version
There exists a global storage version which defines global semantics. This global version typically defines the follow- ing: • What special records exist on the server and what they contain • The payload format of encrypted records on the server • How cryptography of data works Each Sync client is coded to support 1 or more global storage formats. If a client encounters a storage format it doesn’t support, it should probably stop trying to consume data. Under no normal circumstances should a client modify data on a server that is defined with an unknown, newer storage format. Even if an old client wipes all server data and uploads data in its format, newer clients may transparently upgrade to the global storage version they support. Because changing storage formats can cause clients to not be able to use Sync because all clients may not be upgraded to support a newer storage format at the same time, new global storage versions are rarely introduced.
Versions 1, 2, and 3
These were used by an old version of Sync which was deprecated in early 2011. Historical information is available here. These versions should not be in active use and should all be upgraded to a newer storage format.
Version 4
This version initially made the switch to a new crypto model based fully on AES. Because of a faulty implementation of the crypto, version 5 was created to force alpha clients created with the faulty implementation to upgrade. Version 4 and version 5 are therefore practically identical.
Version 5 (Spring 2011 - Current)
Version 5 replaces version 3’s cryptographic model with one based purely on AES. A full overview is available for reference. Historical notes are available.
Version 6 (???)
PROPOSAL In terms of cryptography, version 6 is a natural evolution of version 5. It makes minor changes to low-level cryptogra- phy details. One driving force behind version 6 was the need to support storage of the encrypted Sync Key on the storage server. This was required in order to support BrowserID integration. Another driving force was the transition to version 2.0 of the Storage Service. Strictly speaking, neither of these require a new global storage version. However, they presented an enticing opportu- nity to fix minor issues with version 5. Version 6 is fully documented.
4.3. Sync Client Documentation 159 Mozilla Services Documentation, Release
Collection/Object Format Versions
The formats of unencrypted records stored on the server are also versioned. For example, records in the bookmarks collection are all defined to be of a specific type. Strictly speaking, these versions are tied to a specific global version. However, since all storage formats to date have stored the per-collection version in a special record, these versions in effect apply across all global storage versions. These versions are fully documented.
Global Storage Version 5
This document describes version 5 of Sync’s global storage format. This describes not only the technical details of the storage format, but also some semantics for how clients supporting version 5 should interact with the Sync server.
Overview
A single unencrypted record called the metaglobal record (because it exists in the meta collection with the id global) stores essential data used to instruct clients how to behave. A special record called the cryptokeys record (because it exists in the crypto collection with the id keys) holds encrypted keys which are used to encrypt, decrypt, and verify all other encrypted records on the server.
Cryptography
Overview
Every encrypted record (and all but one record on the server is encrypted) is encrypted using symmetric key encryption and verified using HMAC hashing. The symmetric encryption and HMAC verification keys are only available to client machines: they are not transmitted to the server (at least in any form the server can read). This means that the data on the server cannot be read by anyone with access to the server. The aforementioned symmetric encryption key and and HMAC key constitute what’s called a key bundle. Each key is 256 bits. Individual records are encrypted with AES 256. The encryption key from a key bundle is combined with a per-record 16 byte IV and a user’s data is converted into ciphertext. The ciphertext is signed with the key bundle’s HMAC key. The ciphertext, IV, and HMAC value are then uploaded to the server. When Sync is initially configured, that client generates a random 128 bit sequence called the Sync Key. This private key is used to derive a special key bundle via HKDF. This is called the Sync key bundle. The Sync key bundle is used to encrypt and decrypt a special record on the server which holds more key bundles. Key bundles inside this record are what’s used to encrypt and decrypt all other records on the server.
Terminology
Sync Key 128 bit random value which effectively serves as the master key to Sync. Key Bundle A pair of 256 bit keys. One key is used for symmetric encryption. The other is used for HMAC hashing. Sync Key Bundle A Key Bundle derived from the Sync Key via HKDF. HKDF Cryptographic technique to create values derived from another.
160 Chapter 4. Client Development Mozilla Services Documentation, Release
Bulk Key Bundle A collection of Key Bundles used to secure records. This collection is encrypted with the Sync Key Bundle. Cleartext The plain/clear representation of a piece of data. This is the underlying data that will be exchanged via Sync. It could contain personal and sensitive data. Ciphertext The encrypted version of Cleartext. Ciphertext cannot be turned back into Cleartext without an En- cryption Key. Encryption Key A key in a Key Bundle used for symmetric encryption. This helps turn Cleartext into Ciphertext. HMAC Key A key in a Key Bundle used for HMAC hashing. Symmetric Encryption Process by which Cleartext is converted into Ciphertext and back again with the help of a secret key. HMAC Hashing A technique used to verify that messages (Ciphertexts) haven’t been tampered with. A HMAC Key is applied over a Ciphertext to produce a HMAC Value.
The Sync Key
The Sync Key is the master private key for all of Sync. A single Sync Key is shared between all clients that wish to collaborate with each other using the server. It is important to state that the Sync Key should never be transmitted to an untrusted party or stored where others could access it. This includes inside the storage server. The Sync Key is a randomly generated 128 bit sequence. Generation of this value is left to the client. It is assumed that the chosen random sequence is cryptographically random. For presentation purposes, the Sync Key should be represented as 26 characters from the friendly base32 alphabet with dashes after the 1st, 6th, 11th, 16th, and 21st characters. Our friendly base32 alphabet uses lower case characters and substitutes 8 for l and 9 for o. This prevents ambiguity between 1 and l and 0 and o. In addition, we strip off padding that may be at the end of the string. In pseudo-code: sync_key= randomBytes(16) sync_key_ui= encodeBase32(sync_key).lowerCase().substr(0, 26).replace('l','8').
˓→replace('o','9') sync_key_dashes= sync_key_ui.replaceRegEx(/{.{1,5})/g,"-$1")
Example:
# Generate 16 random bytes \xc7\x1a\xa7\xcb\xd8\xb8\x2a\x8f\xf6\xed\xa5\x5c\x39\x47\x9f\xd2
# Base32 encode Y4NKPS6YXAVI75XNUVODSR472I======
# Lower case and strip y4nkps6yxavi75xnuvodsr472i
# Perform friendly string substitution (note 'o' to '9') y4nkps6yxavi75xnuv9dsr472i
# Add dashes for user presentation y-4nkps-6yxav-i75xn-uv9ds-r472i
4.3. Sync Client Documentation 161 Mozilla Services Documentation, Release
Sync Key Bundle
The Sync Key Bundle is a key bundle derived from the Sync Key via SHA-256 HMAC-based HKDF (RFC 5869). Remember that a key bundle consists of a 256 bit symmetric encryption key and a HMAC key. In pseudo-code: info="identity.mozilla.com/picl/v1/oldsync"
T(1)= HMAC-SHA256(sync_key, info+ 0x01) T(2)= HMAC-SHA256(sync_key, T(1)+ info+ 0x02) encryption_key=T(1) hmac=T(2)
Example: sync_key= \xc7\x1a\xa7\xcb\xd8\xb8\x2a\x8f\xf6\xed\xa5\x5c\x39\x47\x9f\xd2 info="identity.mozilla.com/picl/v1/oldsync"
# Perform HKDF Expansion (1) encryption_key= HKDF-Expand(sync_key, info+" \x01", 32) -> 0x8d0765430ea0d9dbd53c536c6c5c4cb639c093075ef2bd77cd30cf485138b905
# Second round of HKDF hmac= HKDF-Expand(sync_key, encryption_key+ info+" \x02", 32) -> 0xbf9e48ac50a2fcc400ae4d30a58dc6a83a7720c32f58c60fd9d02db16e406216
NB1: The Sync Key is stored in Firefox Accounts. It is referred to as ‘kB’ in https://github.com/mozilla/ fxa-auth-server/wiki/onepw-protocol#-fetching-sync-keys (kA is not used). NB2: In earlier versions, “Sync-AES_256_CBC-HMAC256” + username was used as the hkdf info string instead of “identity.mozilla.com/picl/v1/oldsync”, for instance “[email protected]”.
Record Encryption
Individual records are encrypted using the AES algorithm + HMAC “signing” using keys from a key bundle. You take your cleartext input (which is typically a JSON string representing an object) and feed it into AES. You Base64 encode the raw byte output of that and feed that into HMAC SHA-256. The AES cipher mode is CBC. In pseudo-code: cleartext="SECRET MESSAGE" iv= randomBytes(16) ciphertext= AES256(cleartext, bundle.encryptionKey, iv) hmac= SHA256HMAC(bundle.hmacKey, base64(ciphertext))
Example: encryption_key= 0xd3af449d2dc4b432b8cb5b59d40c8a5fe53b584b16469f5b44828b756ffb6a81 hmac_key= 0x2c5d98092d500a048d09fd01090bd0d3a4861fc8ea2438bd74a8f43be6f47f02 cleartext="SECRET MESSAGE"
162 Chapter 4. Client Development Mozilla Services Documentation, Release
iv= randomBytes(16) -> 0x375a12d6de4ef26b735f6fccfbafff2d ciphertext= AES256(cleartext, encryption_key, iv) -> 0xc1c82acc436de625edf7feca3c9deb4c ciphertext_b64= base64(ciphertext) -> wcgqzENt5iXt9/7KPJ3rTA== hmac= HMACSHA256(hmac_key, ciphertext_b64) -> 0xb5d1479ae2019663d6572b8e8a734e5f06c1602a0cd0becb87ca81501a08fa55
The ciphertext, IV, and HMAC are added to the record and uploaded to the server.
Record Decryption
When you obtain a record, that record will have attached its ciphertext, HMAC, and IV. The client will also have a key bundle (with an encryption key and HMAC key) that is associated with that record’s collection. The first step of decryption is verifying the HMAC. If the locally-computed HMAC does not match the HMAC on the record, the record could either have been tampered with or it could have been encrypted with a different key bundle from the one the client has. Under no circumstances should a client try to decrypt a record if the HMAC verification fails. Once HMAC verification is complete, the client decrypts the ciphertext using the IV from the record and the encryption key from the key bundle. In pseudo-code: ciphertext= record.ciphertext iv= record.iv record_hmac= record.hmac encryption_key= bundle.encryption_key hmac_key= bundle.hmac_key local_hmac= HMACSHA256(hmac_key, base64(ciphertext)) if local_hmac != record_hmac: throw Error("HMAC verification failed.") cleartext= AESDecrypt(ciphertext, encryption_key, iv)
Example:
TODO
New Account Bootstrap
When a new Sync account is initially configured or when an existing Sync account is reset, we perform an initial bootstrap of the cryptographic components. 1. The Sync Key is generated. 2. The Sync key bundle is derived from the Sync Key.
4.3. Sync Client Documentation 163 Mozilla Services Documentation, Release
3. New key bundles are created. 4. The new key bundles are assembled into a bulk key bundle/record and uploaded to the server after being encrypted by the Sync key bundle. At this point, the client is bootstrapped from a cryptography perspective.
Metaglobal Record
The meta/global record is a special record on the Sync Server that contains general metadata to describe the state of data on the Sync Server. This state includes things like the global storage version and the set of available engines/collections on the server. The meta/global record is different from other records in that it is not encrypted. The payload of this record is a JSON string that deserializes to an object (i.e. a hash). This object has the following fields: • storageVersion: Integer version of the global storage format used • syncID: Opaque string that changes when drastic changes happen to the overall data. Change of this string can cause clients to drop cached data. The Firefox client uses 12 randomly generated base64url characters, much like for WBO IDs. • engines: A hash with fields of engine names and values of objects that contain version and syncID fields, which behave like the storageVersion and syncID fields on this record, but on a per-engine level. In Protocol 1.5, an additional field is present: • declined: engines that are not present in engines, and are not present in this array, can be presumed to be neither enabled nor explicitly declined. If a user has explicitly declined an engine, rather than e.g., not having the option due to missing functionality on the client, then it should be added to this list in the uploaded meta/global record. No engine should be present in both engines and declined; if an error results in this situation, engines takes precedent.
Example
{ "syncID":"7vO3Zcdu6V4I", "storageVersion":5, "engines":{ "clients":{"version":1,"syncID":"Re1DKzUQE2jt"}, "bookmarks":{"version":2,"syncID":"ApPN6v8VY42s"}, "forms":{"version":1,"syncID":"lLnCTaQM3SPR"}, "tabs":{"version":1,"syncID":"G1nU87H-7jdl"}, "history":{"version":1,"syncID":"9Tvy_Vlb44b2"}, "prefs":{"version":2,"syncID":"8eONx16GXAlp"} }, "declined":["passwords"] }
Semantics and Behavior
Clients should fetch the metaglobal record after it has been determined that a full sync should be performed. If the metaglobal record does not exist, the client should issue a request to delete all data from the server and then create and upload a new metaglobal record.
164 Chapter 4. Client Development Mozilla Services Documentation, Release
In the common scenario where the metaglobal record exists, the client should first check that the storage version from the record is supported. If it is, great. If the storage version is older than what the client supports, the client may choose to upgrade server data to a new storage version. Keep in mind this may break older clients! If the storage version is newer than what the client supports, all bets are off and the client should infer that a new version is available and that the user should upgrade. Clients should not modify any data on a server if the global storage version is newer than what is supported.
crypto/keys record
In storage version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.
Encrypting and decrypting
The `crypto/keys` WBO is encrypted and verified just like any other WBO, except the Sync Key bundle is used instead of a bulk key bundle.
Format
The inner payload of the crypto/keys record contains the following fields: • default: Array of length 2 containing the default key pair (encryption key, HMAC key). • collections: Object mapping collection name to collection-specific key pairs which are arrays of length 2 (en- cryption key, HMAC key). • collection: String stating the collection of the record. Currently fixed to “crypto”. Each key is Base64 encoded.
Example
{"id":"keys", "collection":"crypto", "collections":{}, "default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=', 'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}
Collection Records
All records in non-special collections have a common payload format. The payload is defined as the JSON encoding of an object containing the following fields: • ciphertext: Base64 of encrypted cleartext for underlying payload. • IV: Base64 encoding of IV used for encryption. • hmac: Base64 encoding of HMAC for this message. Here is an example:
4.3. Sync Client Documentation 165 Mozilla Services Documentation, Release
{ "payload":"{ \"ciphertext\":\
˓→"K5JZc7t4R2DzL6nanW+xsJMDhMZkiyRnG3ahpuz61hmFrDZu7DbsYHD77r5Eadlj\",\"IV\":\
˓→"THPKCzWVX35\\/5123ho6mJQ==\",\"hmac\":\
˓→"78ecf07c46b12ab71b769532f15977129d5fc0c121ac261bf4dda88b3329f6bd\"}", "id":"GJN0ojnlXXhU", "modified": 1332402035.78 }
The format of the unencrypted ciphertext is defined by the collection it resides in. See the Object Formats documen- tation for specifics. That being said, the cleartext is almost certainly a JSON string representing an object. This will be assumed for the examples below.
Encryption
Let’s assume you have the following JSON payload to encrypt:
{ "foo":"supersecret", "bar":"anothersecret" }
Now, in pseudo-code:
# collection_name is the name of the collection this record will be inserted # into. bulk_key_bundle is an object that represents the decrypted # crypto/keys record. The called function simply extracts the appropriate # key pair for the specified collection. key_pair= bulk_key_bundle.getKeyPair(collection_name);
# Just some simple aliasing. encryption_key= key_pair.encryption_key hmac_key= key_pair.hmac_key iv= randomBytes(16)
# cleartext is the example JSON above. ciphertext= AES256(cleartext, encryption_key, iv) ciphertext_b64= Base64Encode(ciphertext) hmac= HMACSHA256(ciphertext_b64, hmac_key) payload={ "ciphertext": ciphertext_b64, "IV": Base64Encode(iv), "hmac": Base64Encode(hmac) } record.payload= JSONEncode(payload)
Decryption
Decryption is just the opposite of encryption. Let’s assume we get a record from the server:
166 Chapter 4. Client Development Mozilla Services Documentation, Release
{ "payload":"{ \"ciphertext\":\
˓→"K5JZc7t4R2DzL6nanW+xsJMDhMZkiyRnG3ahpuz61hmFrDZu7DbsYHD77r5Eadlj\",\"IV\":\
˓→"THPKCzWVX35\\/5123ho6mJQ==\",\"hmac\":\
˓→"78ecf07c46b12ab71b769532f15977129d5fc0c121ac261bf4dda88b3329f6bd\"}", "id":"GJN0ojnlXXhU", "modified": 1332402035.78 }
To decrypt it:
fields= JSONDecode(record.payload)
# The HMAC is computed over the Base64 version of the ciphertext, so we # leave the encoding intact for now. ciphertext_b64= fields.ciphertext
remote_hmac= Base64Decode(fields.hmac) iv= Base64Decode(fields.IV)
key_pair= bulk_key_bundle.getKeyPair(collection_name) encryption_key= key_pair.encryption_key hmac_key= key_pair.hmac_key
local_hmac= HMACSHA256(ciphertext_b64, hmac_key)
if local_hmac != remote_hmac: throw Error("HMAC verification failed.")
ciphertext= Base64Decode(ciphertext_b64)
cleartext= AESDecrypt(ciphertext, encryption_key, iv)
object= JSONDecode(cleartext)
Global Storage Version 6
Attention: This document is a proposal. It will likely change.
This document describes version 6 of Sync’s global storage format. This describes not only the technical details of the storage format, but also some semantics for how clients supporting version 6 should interact with the Sync server.
Cryptographic Model
All data on the server (with the exception of a single record containing non-private metadata used to help clients perform sanity checking) is encrypted on the client using AES symmetric encryption with 256 bit keys. Encrypted data is secured against tampering by employing HMAC-SHA256 hashing. The cryptographic model frequently relies on pairs of 256 bit keys. One key is used for AES symmetric encryption; the other for HMAC verification. We refer to such a pair of keys as a Key Bundle. Data on the server is organized into collections (e.g. history, bookmarks). Every collection has a single Key Bundle associated with it. We refer to a Key Bundle that is affiliated with a collection as a Collection Key Bundle. A single
4.3. Sync Client Documentation 167 Mozilla Services Documentation, Release
Collection Key Bundle is used to perform cryptographic operations on every record in the collection to which it is associated. It is recommended, but not technically required, that each Collection Key Bundle be associated with at most a single collection. Special records on the server hold mappings of collection names to their respective Collection Key Bundles. The Collection Key Bundles are encrypted using another higher-level Key Bundle before they are stored on the server. We refer to these higher-level Key Bundles as Key-Encrypting Key Bundles. And, the entity that holds the mapping to Collection Key Bundles is referred to as a crypto record (because it is a record stored in the crypto collection). In the simple case, we have a single Key-Encrypting Key Bundle used to encrypt the collection of all Collection Key Bundles. Each Collection Key Bundle is used to encrypt every record in the collection to which it is associated. In other words, we have a master key used to unlock other keys which in turn unlock data on the server. In graph form:
Key-Encrypting Key Bundle
Bookmarks Collection Key Bundle History Collection Key Bundle
Bookmark A Bookmark B History 1 History 2
This specification establishes no rules for which crypto records exist or for how Key-Encrypting Key Bundles are managed. This is entirely up to the client. In other words, key management is a convention between clients. If you are interested in interoperating with Firefox Sync, see Mozilla’s Sync Service. This is the essence of the cryptographic model. More details are explained below.
Representation of Key Pairs
While Key Bundles consist of two separate keys, they should be thought of as a single immutable entity. To enforce this, a Sync Key Bundle is represented as a single blob of data. The blob consists of the 256 bits of the encryption key followed by the 256 bits of the HMAC key followed by optional metadata. The format of the metadata is not currently defined. If no metadata is present, no extra information is recorded and the Key Bundle is represented as a single 512 bit (64 byte) blob. In pseudo-code:
key_bundle= encryption_key+ hmac_key+ metadata
When encoded in JSON, key bundles are Base64 encoded. Note that keys are not stored in plaintext, so the Base64 encoding will apply to the ciphertext. See below.
168 Chapter 4. Client Development Mozilla Services Documentation, Release
Encrypted Records
Most encrypted records share a common payload format and method for encryption and decryption. The payload of an encrypted records effectively consists of the following fields: • ciphertext: The encrypted version of the underlying data. • IV: Initialization vector used by AES encryption. • HMAC: HMAC for the encrypted message. Since these 3 items are all related and all are needed to decrypt and verify individual messages, they are represented by a single entity - a buffer containing all 3 fields concatenated together. Each binary buffer holds the raw bytes constituting the HMAC signature, followed by the raw bytes of the IV, followed by the raw bytes of the ciphertext. In pseudo-code:
data= hmac+ iv+ ciphertext
The HMAC signature is always the length of the HMAC key. Since Sync uses 256-bit HMAC keys, the HMAC signature is 256 bits, or 32 bytes. The IV is fixed-width at 16 bytes. The ciphertext is variable length. We refer to this 3-tuple of encryption matter as Encrypted Data. When represented in JSON, the raw bytes constituting the Encrypted Data are Base64 encoded.
Encryption
Encryption is the process of taking some piece of data, referred to as cleartext, and converting it to Encrypted Data. We start with a Key Bundle and cleartext. In pseudo-code:
# collection_name is the name of the collection this record will be inserted # into. The called function obtains the appropriate key bundle depending on # the destination collection of the record. bundle= getBundleForCollection(collection_name)
# Just some aliasing for readability. encryption_key= bundle.encryption_key hmac_key= bundle.hmac_key
iv= randomBytes(16)
ciphertext= AES256Encrypt(encryption_key, iv, cleartext)
# Now compute the HMAC. Be sure to include the IV in the computation. message= iv+ ciphertext hmac= HMACSHA256(hmac_key, message)
encrypted_data= hmac+ message
4.3. Sync Client Documentation 169 Mozilla Services Documentation, Release
# When going to JSON, the binary payload buffer is Base64-encoded first. record.payload= Base64Encode(encrypted_data)
Decryption
Decryption is the process of taking Encryted Data and turning it into cleartext. Decryption requires Encrypted Data and a Key Bundle. In pseudo-code: bundle= getBundleForCollection(collection_name) encryption_key= bundle.encryption_key hmac_key= bundle.hmac_key
# If grabbing the record from JSON, it will Base64 encoded. payload_b64= record.payload encrypted_data= Base64Decode(payload_b64)
# HMAC is first 32 bytes of payload. hmac_remote= encrypted_data[0:31]
# IV is the 16 bytes after the HMAC iv= encrypted_data[32:47]
# ciphertext is everything that's left. ciphertext= encrypted_data[48:]
# The first step of decryption is verifying the HMAC. The HMAC is computed # over both the IV and the ciphertext. hmac_local= HMACSHA256(hmac_key, iv+ ciphertext) if hmac_local != hmac_remote: throw new Error("HMAC verification failed!"); cleartext= AESDecrypt(encryption_key, iv, ciphertext)
Global Metadata Record
The meta/global record exists with the same semantics as version 5, the only difference being that the storageVersion is 6 and the engines key has been renamed to repositories. TODO carry version 5’s documentation forward. Example:
{ "syncID":"7vO3Zcdu6V4I", "storageVersion":6, "repositories":{ "clients":{"version":1,"syncID":"Re1DKzUQE2jt"}, "bookmarks":{"version":2,"syncID":"ApPN6v8VY42s"}, "forms":{"version":1,"syncID":"lLnCTaQM3SPR"}, "tabs":{"version":1,"syncID":"G1nU87H-7jdl"}, "history":{"version":1,"syncID":"9Tvy_Vlb44b2"}, "passwords":{"version":1,"syncID":"yfBi2v7PpFO2"},
170 Chapter 4. Client Development Mozilla Services Documentation, Release
"prefs":{"version":2,"syncID":"8eONx16GXAlp"} } }
crypto Collection
There exists a special collection on the server named crypto. This collection holds records that contain mappings of collections to Collection Key Bundles. Each record in the crypto collection has associated with it specific semantics. This specification is intentionally vague as to what records and semantics are defined, as it is up to clients to define those. In other words, the set of records on the server and the specifics of which Collection Key Bundles they contain and/or which Key Encrypting Key Bundle is used to secure them is left to the purview of the client. The rationale for this is that users may wish to manage their Collection Key Bundles with different levels of access or security. For example, the record containing all the keys may only be decrypted with a highly secure parent key, while another record may contain keys for less-sensitive collections, which can be unlocked using a key derived from a less secure method, such as PBKDF2. Clients should support the ability to intelligently use different sets of Collection Key Bundles, depending on what the user has provided them access to. This means clients should not be eager to delete collections for which it doesn’t have the Collection Key Bundle, as the user may have purposefully withheld access to that specific collection.
Record Format
The exact format of records in this collection has yet to be decided. We have a few options.
Option 1
The payload of every record is an object containing the following fields: • collections - (required) A cleartext wrapping of collection names to Encrypted Data. The decrypted values are Key Bundles used to encrypt the collection to which it is tied. • encryptingKey - (optional) An encrypted* Key Bundle used to encrypt other encrypted data in this record. For example:
{ "collections":{ "bookmarks":"ENCRYPTED KEY 0", "history":"ENCRYPTED KEY 1" }, "encryptingKey":"ENCRYPTED KEY-ENCRYPTING KEY" }
The client would – if not delivered out-of-band – decrypt the encrypting key. This would require its parent key and the contents of this record. The client would then take the decrypted key-encrypting key and decrypt the individual Collection Key Bundles. Pros: • Simple Cons:
4.3. Sync Client Documentation 171 Mozilla Services Documentation, Release
• Server data reveals which encrypting keys can be used to unlock which collections.
Option 2
This is similar to Option 1 except that the mapping info is itself encrypted. For example:
{ "data":"ENCRYPTED DATA", "encryptingKey":"ENCRYPTED KEY-ENCRYPTING KEY" }
The decrypted key encrypting key would first decrypt the data field. This would expose the mapping of collection names to encrypted Key Bundles, just as in Option 1. From there, the same key-encrypting key would decrypt each individual Key Bundle. Yes, the Key Bundles are encrypted with the same key twice. We do not want the Key Bundles unencrypted after the first unwrapping because we want clients to be designed such that they never have to touch unencrypted key matter. In the case of Firefox, this means Sync can operate in FIPS mode since NSS will be the only entity handling the unencrypted keys. Pros: • Server data does not reveal which keys can unlock which collections Cons: • More complicated than version 1 • Double encryption involves extra work.
No encryptingKey Variation
There is a variation of the above options where the encrypted key encrypting key is not stored in the record. Instead, it is stored in another record on the server or not stored on the server at all. These variations differ only in the specifics of the record payload.
Changes Since Version 5
Sync Keys Consolidated
The Sync Key has traditionally been 128 bits (often encoded as 26 “friendly” Base32 characters). The historical reason for it being 128 bits is that in early versions of Sync (before J-PAKE), people would need to manually enter the Sync Key to pair other devices. Even with J-PAKE, people may need to manually enter the Sync Key (known as the Recovery Key in UI parlance) into their client. From the 128-bit Sync Key, two 256-bit keys were derived via HKDF. With the intent to use BrowserID’s key wrapping facility, we feel Sync no longer has the requirement that the Sync Key be manageable to enter from UI. This is because your Sync Key will be accessible merely by logging into BrowserID (your BrowserID credentials will unlock a BrowserID user key and that user key can unwrap an encrypted Sync Key stored on the server). (We expect that users not using BrowserID will use some other mechanism for key exchange other than keyboard entry.)
172 Chapter 4. Client Development Mozilla Services Documentation, Release
Therefore, in version 6, the Sync Key will consist of a pair of 256-bit keys. Each key will be generated from a cryptographically secure random number generator and will not be derived from any other source. This effectively replaces the single 128-bit random key and two 256-bit HKDF-derived keys with two completely random 256-bit keys.
Sync Key Stored on Server
Version 6 supports storing the encrypted Sync Key on the Storage Server.
Key Pair Encoding
In version 5, key pairs (the two 256-bit keys used for symmetric encryption and HMAC verification) were represented in payloads as arrays consisting of two strings, each representing the Base64 encoded version of the key. In version 6, key pairs are transmitted as a a single string or byte array. The two keys are merely concatenated together to form one 512-bit data chunk. Version 6 also supports additional metadata after the keys. However, the format of this metadata is not yet defined.
IV Included in HMAC Hash
In version 6, the IV is included in the HMAC hash. In previous versions, the IV was not included. This change adds more theoretical security to the verification process.
HMAC Performed Over Raw Ciphertext
In version 6, the HMAC is performed over the raw ciphertext bytes. In version 5, HMAC was performed over the Base64 encoding of the ciphertext.
Representation of Crypto Fields in Records
In version 6, the representation of cryptographic fields has been hidden from the record payload. In version 5, the payload of encrypted records was the Base64 encoding of the JSON encoding of an object with the fields ciphertext, hmac, and IV. In version 6, we embed all 3 elements in one opaque field. While the client will need to know how to extract the individual cryptographic components, the transport layer happily deals with a single string of bytes. In the case of JSON encoding, the payload is now the Base64 representation of the single string, not a JSON string.
Requiring Storing Separate Key Pairs for Collections
Version 6 requires that separate Key Bundles be used for each collection. The previous version had a default Key Bundle that could be used to decrypt multiple collections. Clients would look for a collection-specific key in the crypto/keys record then fall back to the default. In practice, clients (notably Firefox), did not generate multiple keys by default. Version 6 is dropping support for the default key and requiring that each collection use a separate key. This change is being made in an effort to be forward compatible with future data recovery and sharing scenarios. The requirement of separate keys per collections effectively requires an extra link in the crypto chain where extra functionality can be inserted for one collection without impacting other collections.
4.3. Sync Client Documentation 173 Mozilla Services Documentation, Release
Metaglobal Record Format Change
The engines key in the metaglobal record has been renamed to repositories. Semantics are preserved.
Firefox object formats
Decrypted data objects are cleartext JSON strings. Each collection can have its own object structure. This page documents the format of each collection. The object structure is versioned with the version metadata stored in the meta/global payload. The following sections, named by the corresponding collection name, describe the various object formats and how they’re used. Note that object structures may change in the future and may not be backwards compatible. In addition to these custom collection object structures, the Encrypted DataObject adds fields like id and deleted. Also, remember that there is data at the Weave Basic Object (WBO) level as well as id, modified, sortindex and payload.
Add-ons
Version 1
Version 1 is likely only affiliated with storage format 5 clients. • addonID string: Public identifier of add-on. This is the id attribute from an Addon object obtained from the AddonManager. • applicationID string: The application ID the add-on belongs to. • enabled bool: Indicates whether the add-on is enabled or disabled. true means enabled. • source string: Where the add-on came from. amo means it came from addons.mozilla.org or a trusted site.
Bookmarks
Version 1
One bookmark record exists for each bookmark item, where an item may actually be a folder or a separator. Each item will have a type that determines what other fields are available in the object. The following sections describe the object format for a given type. Each bookmark item has a parentid and predecessorid to form a structure like a tree of linked-lists to provide a hierarchical ordered list of bookmarks, folders, etc. bookmark
This describes a regular bookmark that users can click to view a page. • title string: name of the bookmark • bmkUri string uri of the page to load • description string: extra description if provided • loadInSidebar boolean: true if the bookmark should load in the sidebar • tags array of strings: tags for the bookmark
174 Chapter 4. Client Development Mozilla Services Documentation, Release
• keyword string: alias to activate the bookmark from the location bar • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first) • type string: “bookmark”
microsummary
Microsummaries allow pages to be summarized for viewing from the toolbar. This extends bookmark, so the usual bookmark fields apply. • generatorUri string: uri that generates the summary • staticTitle string: title to show when no summaries are available • title string: name of the microsummary • bmkUri string uri of the page to load • description string: extra description if provided • loadInSidebar boolean: true if the bookmark should load in the sidebar • tags array of strings: tags for the bookmark • keyword string: alias to activate the bookmark from the location bar • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first) • type string: “microsummary” query
Place queries are special bookmarks with a place: uri that links to an existing folder/tag. This extends bookmark, so the usual bookmark fields apply. • folderName string: name of the folder/tag to link to • queryId string (optional): identifier of the smart bookmark query • title string: name of the query • bmkUri string place: uri query • description string: extra description if provided • loadInSidebar boolean: true if the bookmark should load in the sidebar • tags array of strings: tags for the query • keyword string: alias to activate the bookmark from the location bar • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first)
4.3. Sync Client Documentation 175 Mozilla Services Documentation, Release
• type string: “query”
folder
Folders contain bookmark items like bookmarks and other folders. • title string: name of the folder • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first) • type string: “folder”
livemark
Livemarks act like folders with a dynamic list bookmarks, e.g., a RSS feed. This extends folder, so the usual folder fields apply. • siteUri string: site associated with the livemark • feedUri string: feed to get items for the livemark • title string: name of the livemark • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first) • type string: “livemark”
separator
Separators help split sections of a folder. • pos string: position (index) of the separator • parentid string: GUID of the containing folder • parentName string: name of the containing folder • predecessorid string: GUID of the item before this (empty if it’s first) • type string: “separator”
Version 2
Same as engine version 1, except: • the predecessorid is removed from all records, • instead folder and livemark records have a children attribute which is an array of child GUIDs in order of their appearance in the folder: • children array of strings: ordered list of child GUIDs
176 Chapter 4. Client Development Mozilla Services Documentation, Release
• the special folders ‘menu’ and ‘toolbar’ now have records that are synced, purely to maintain order within them according to their ‘’‘children’‘’ array.
Version 3
Note: Proposal corresponding with storage format 6.
Same as version 2 except: • Support for microsummaries is removed • We use the ASCII URL TODO document full format here since diffs are inconvenient to read.
Clients
Version 1
Client records identify a user’s one or multiple clients that are accessing the data. The existence of client records can change the behavior of the Firefox Sync client – multiple clients and/or mobile clients result in syncs to happen more frequently. • name string: name of the client connecting • type string: type of the client: “desktop” or “mobile” • commands array: commands to be executed upon next sync In Protocol 1.5, client records additionally include: • version string: a version indicator for this client, such as “29.0a1”. Optional. • protocols array: an array of Sync protocol versions supported by this client, such as [”1.1”, “1.5”]. Optional. In Bug 1097222 additional optional fields were added: • os string: an OS name, most likely one of “Darwin” (Mac OS X), “WINNT” (Windows), “Android”, or “iOS”. • appPackage string: an unambiguous identifier for the client application. For Android, this is the package (e.g., org.mozilla.firefox_beta). For desktop this is the value of Services.appinfo.ID. • application string: a human-readable application name, such as “Nightly” or “Firefox”. • formfactor string: a value such as “phone”, “tablet” (or the more specific “largetablet”, “smalltablet”), “desk- top”, “laptop”, “tv”. • device string: a description of the hardware that this client uses. Currently only supported by Android; returns values like “HTC One”. If these fields are missing, clients are expected to fall back to behaviors that do not depend on the missing data. Clients should preserve existing fields if possible when sending commands to another client.
4.3. Sync Client Documentation 177 Mozilla Services Documentation, Release
Version 2 (never deployed)
Note: Proposal corresponding with storage format 6.
Each client has its own record which it is authoritative for. No other client should modify another client’s record except in the case where records are deleted. The payload of a client record has the following fields: • name string: The name of the client. This is a user-facing value and may be provided by the user. • formfactor string: The form factor of the client. Recognized values include phone, tablet, laptop, desktop. • application string: String identifying the application behind the client. This should only be used for presentation purposes (e.g. choosing what logo to display). • version string: The version of the client. This is typically the version of the application. Again, this should only be used for presentation purposes. • capabilities object: Denotes the capabilities a client possesses. Keys are string capability names. Values are booleans indicating whether the capability is enabled. Modifying the capabilities of another client’s record should not change the enabled state on that client. • mpEnabled bool: Whether master password is enabled on the client. If master password is enabled on any client in an account, the current client should hesitate before downloading passwords if master password is not enabled locally, as this would decrease the security of the passwords locally since they wouldn’t be protected with a master password.
Commands
Version 1
Note: Proposal corresponding with storage format 6.
This collection holds commands for clients to process. The ID of command records is randomly generated. Command records contain an extra unencrypted field in the BSO that says which client ID they belong to. The value is the hash of the client ID with the commands engine salt. Command data is an object with the following fields: • receiverID string: Client ID of the client that should receive the command. This is duplicated inside the payload so it can be verified by the HMAC. • senderID string: Client ID of the client that sent the command. • created number: Integer seconds since Unix epoch that command was created. • action string: The action to be performed by the command. Each command has its own name that uniquely identifies it. It is recommended that actions be namespaced using colon-delimited notation. Sync’s commands are all prefixed with sync:. e.g. sync:wipe. If a command is versioned, the name is the appropriate place to convey that versioning. • data object: Additional data associated with command. This is dependent on the specific command type being issued.
178 Chapter 4. Client Development Mozilla Services Documentation, Release
Forms
Form data is used to give suggestions for autocomplete for a HTML text input form. One record is created for each form entry. • name string: name of the HTML input field • value string: value to suggest for the input
History
Version 1
Every page a user visits generates a history item/page. One history (page) per record. • histUri string: uri of the page • title string: title of the page • visits array of objects: a number of how and when the page was visited • date integer: datetime of the visit • type integer: transition type of the visit
Version 2 (never deployed)
Note: Proposal corresponding with storage format 6.
History visits are now stored as a timeline/stream of visits. The historical information for a particular site/URL is spread out of N>=1 records. Payloads have the structure:
{ "items":[ "uri":"http://www.mozilla.org/", "title":"Mozilla", "visits":{ 1:[1340757179.82, 184], 2:[1340341244.31, 12,4] } ] }
The bulk of the payload is a list of history items. Each item is both a place and a set of visits. • uri string: URI of the page that was visited. • title string: Title of the page that was visited. • visits object: Mapping of visit type to visit times. The keys in visits define the transition type for the visit. They can be one of the following: • 1: A link was followed. • 2: The URL was typed by the user.
4.3. Sync Client Documentation 179 Mozilla Services Documentation, Release
• 3: The user followed a bookmark. • 4: Some inner content was loaded. • 5: A permanent redirect was followed. • 6: A temporary redirect was followed. • 7: The URL was downloaded. • 8: User follows a link that was in a frame. These correspond to nsINavHistoryService’s transition type constants
[100000000.000, 10.100, 5.200]
Correspond to the times:
100000000.000 100000010.100 100000015.300
The use of deltas to represent times is to minimize the serialized size of visits.
Passwords
Saved passwords help users get back into websites that require a login such as HTML input/password fields or HTTP auth. • hostname string: hostname that password is applicable at • formSubmitURL string: submission url (GET/POST url set by