WebExtensions support on AMO

It’s been possible to submit WebExtensions add-ons to addons.mozilla.org (AMO) since February, but I wanted to take a moment to highlight some of the improvements we’ve made since then that make it much easier to get your Chrome or Opera add-on into AMO.

These features have all been deployed to AMO over the last couple of months and are available for use.

New linter

We are now using the add-ons linter for WebExtensions which means that when we parse a WebExtensions add-on, we are running through a new JavaScript-powered linter, instead of the Python one. This parsing occurs each time you upload a new add-on or a new version of an add-on.

It uses eslint to parse the JavaScript and a schema to verify that the WebExtensions manifest is correct. Because we’ve started to implement functionality in the new add-ons linter that didn’t exist in the old version, more things will be showing up as warnings.

As an example, if you request a permission in the manifest that Firefox doesn’t support, you’ll get a warning:

image00

Relaxed upload criteria

If you’ve developed and add-on for Chrome or Opera, we’d like you to upload it to AMO. For this reason, we’ve relaxed a few criteria around AMO. Once you’ve verified that an add-on works correctly using about:debugging, you should then be able to upload it to AMO quite easily. We’ve hopefully made this easier through the following steps:

Optional add-on id

If your WebExtension does not have an id specified in the manifest.json, then AMO will generate an id for your add-on and add it to the add-on signing process. You can then get the add-on id from AMO and re-use that in later API calls, if needed.

The command line tool web-ext also supports optional ids.

Uploads

AMO will now accept an add-on as a zip file or a crx file. It will still only emit xpi files that Firefox and other Mozilla projects can consume. In the case of a crx file, the extra data added by the Chrome store will be stripped out of the file.

We recommend using a tool like web-ext to generate the appropriate file for your add-on.

Comments in the manifest file

Although comments are not valid in a JSON file, Chrome supports comments in the manifest.json file. To keep compatibility with Chrome, the addons-linter and hence AMO now support comments in the manifest file, but only comments starting with a double slash (//).

Translations

Finally, AMO now supports translations in the manifest. This again follows the Chrome standard, so that:

//in manifest.json:
"default_locale": "en",
"name": "__MSG_appName__",

//in _locales/en/messages.json:
{
  "appName": {
    "message": "My Add-on",
    "description": "The name of the add-on."
  }
}

//in _locales/de/messages.json:
{
  "appName": {
    "message": "Mein Add-on"
  }
}

The manifest will be parsed and the translations automatically entered into AMO. You need to ensure that default_locale is set in the manifest for the localisations to be activated. On the first upload, AMO will process the translations:

Screenshot 2016-07-14 09.44.11

If you’d like to get involved then join our mailing list or check out our repositories on Github.

Mozlondon Debriefer

teamFrom June 13 – 17, more than a thousand Mozillians descended on London to celebrate the accomplishments of the past six months and collaborate on priorities for the latter half of the year. The add-ons team and a handful of volunteers did just that, and we wanted to share some of the highlights.

Add-on reviews

We celebrated the fact that review times have dropped dramatically in 2016. Last year, most add-on submissions got clogged in the review queue for 10 or more days. Now, 95% of submissions are reviewed in under five days (84% in less than 24 hours!). This is a huge boon to our developer community and a great credit to our tireless, mostly-volunteer review team.

Review efficacy should continue to trend positively over the balance of this year. We furthered plans to simplify the review process by eliminating preliminary reviews, among other streamlining initiatives.

Engineering

It’s likely not a surprise to learn that WebExtensions dominated engineering discussions in London. Here are a few top-level items…

… A WebExtensions Advisory Group has been formed. This cross-disciplinary body is primarily tasked with providing input around the development of future APIs.

… We saw some inspired demos, like runtime.connectNative, which will allow add-ons to communicate out of process; and the auto-reload function for WebExtensions (set to land in 49); plus a demo of a hybrid add-on that allows an SDK add-on to embed a WebExtension add-on, allowing developers to start using the WebExtension API and migrate their users.

… Prioritized future work for WebExtensions was solidified, including permissions support, Chrome parity support for top Chrome content, and end-to-end automated installation testing.

… We recapped many recent WebExtensions advancements, such as the implementation of dozens of new APIs (e.g. history, commands, downloads, webNavigation, etc.)

Community & Editorial

It’s worth recognizing all the diverse ways our community contributes to the success of add-ons—coders, content reviewers, curators, translators, tech writers, evangelists, student ambassadors, and more—our community is integral in providing Firefox users with the power to personalize their Web experience and AMO’s ability to support a massive ecosystem of more than 32,000 extensions and 400,000 themes.

One of the focal points of our community discussions in London was: How can we do more to move volunteers up the contribution curve, so we can nurture the next generation of leaders? What more can we do to turn bug commenters into bug fixers, developers into evangelists, and so forth? Check out the notes from this session and add your ideas there!

On the editorial side, we started exploring ways of re-thinking content discovery. Of course we have the AMO site as a content discovery destination with more than 15 million monthly users, but research tells us the majority of Firefox users hardly know what an add-on is, let alone where to find one. So how can we introduce the wonder of add-ons to a broader Firefox audience?

A general consensus emerged—we need to take the content to where the user is, to key moments in their everyday Web experience where perhaps an add-on can address a need. This is just one of the projects we’ll begin tackling in the second half of the year. If you’d like to get involved, you’re invited to join our weekly public UX meeting.

To get involved with the add-ons world, please visit our wiki page, or get in touch with us at irc.mozilla.org in the #addons channel.

Multi-process Firefox and AMO

In Firefox 48, which reaches the release channel on August 1, 2016, mullti-process support (code name “Electrolysis”, or “e10s”) will begin rolling out to Firefox users without any add-ons installed.

In preparation for the wider roll-out to users with add-ons installed, we have implemented compatibility checks on all add-ons uploaded to addons.mozilla.org (AMO).

There are currently three possible states:

  1. The add-on is a WebExtension and hence compatible.
  2. The add-on has marked itself in the install.rdf as multi-process compatible.
  3. The add-on has not marked itself compatible, so the state is currently unknown.

If a new add-on or a new version of an old add-on is not multi-process compatible, a warning will be shown in the validation step. Here is an example:

image01

In future releases, this warning might become more severe as the feature nears full deployment.

For add-ons that fall into the third category, we might implement a more detailed check in a future release, to provide developers with more insight into the “unknown” state.

After an add-on is uploaded, the state is shown in the Developer Hub. Here is an example:

image00

Once you verify that your add-on is compatible, be sure to mark it as such and upload a new version to AMO. There is documentation on MDN on how to test and mark your add-on.

If your add-on is not compatible, please head to our resource center where you will find information on how to update it and where to get help. We’re here to support you!

WebExtensions for Firefox 49

Firefox

Firefox 49 landed in Developer Edition this week, so we have another update on WebExtensions for you!

Since the release of Firefox 48, we feel WebExtensions are in a stable state. We recommend developers start to use the WebExtensions API for their add-on development. Since the last release, more than 35 bugs were closed on WebExtensions alone.

If you have authored an add-on in the past and are curious about how it’s affected by the upcoming changes, please use this lookup tool. There is also a wiki page and MDN articles filled with resources to support you through the changes.

APIs Implemented

The history API allows you to interact with the browser history. In Firefox 49 the APIs to add, query, and delete browser history have been merged. This is ideal for developers who want to build add-ons to manage user privacy.

In Firefox 48, Android support was merged, and in Firefox 49 support for some of the native elements has started to land. Firefox 49 on Android supports some of the pageAction APIs. This work lays the foundation for new APIs such as tabs, windows, browserAction in Android.

The WebNavigation API now supports the manual_subframe transitionType and keeps track of user interaction with the url bar appropriately. The downloads API now lets you download a blob created in a background script.

For a full list of bugs, please check out Bugzilla.

In progress

Things are a little bit quieter recently because there are things in progress that have absorbed a lot of developer time. They won’t land in the tree for Firefox 49, but we’ll keep you updated on their progress in later releases.

storage.sync

This API allows you to store some data for an add-on in Firefox and have it synced to another Firefox browser. It is most commonly used for storing add-on preferences, because it is not designed to be a robust data storage and syncing tool. For sync, we will use Firefox Accounts to authenticate users and enforce quota limits.

Whilst the API contains the word “sync” and it uses Firefox Accounts, it should be noted that it is different from Firefox Sync. In Firefox Sync there is an attempt to merge data and resolve conflicts. There is no similar logic in this API.

You can track the progress of storage.sync in Bugzilla.

runtime.connectNative

This API allows you to communicate with other processes on the host’s operating system. It’s a commonly used API for password managers and security software which needs to communicate with external processes.

To communicate with a native process, there’s a two-step process. First, your installer needs to install a JSON manifest file at an appropriate file location on the target computer. That JSON manifest provides the link between Firefox and the process. Secondly, the user installs the add-on. Then the add-on can call the connectNative, sendNativeMessage and other APIs:

chrome.runtime.sendNativeMessage('your-application',
  { text: "Hello" },
  function(response) {
    console.log("Received " + response);
});

Firefox will start the process if it hasn’t started already, and pipe commands through to the process. Follow along with the progress of runtime.connectNative on Bugzilla.

WebExtensions transition

With these ongoing improvements, we realise there are lots of add-ons that might want to start moving towards WebExtensions and utilise the new APIs.

To allow this, you will soon be able to embed a WebExtension inside an add-on. This allows you to message the WebExtension add-on.

The following example works with SDK add-ons, but this should work with any bootstrapped add-on. Inside your SDK add-on you’d have a directory called webextensions containing a full-fledged WebExtension. In the background page of that WebExtension will be the following:

chrome.runtime.sendMessage("test message", (reply) => {
  console.log("embedded webext got a reply", reply);
});

Then you’d be able to reply in the SDK add-on:

var { api } = require('sdk/webextension');
api.onMessage.addListener((msg, sender, sendReply) =>
  console.log("SDK add-on got a message", {msg,sender}); 
  sendReply("reply");
});

This demonstrates sending a message from the WebExtension to the SDK add-on.Persistent bi-directional ports will also be available.

Using this technique, we hope add-on developers can leverage WebExtensions APIs as they start migrating their add-on over to WebExtensions. Follow along with the progress of communication on Bugzilla.

There are also lots of other ways to get involved with WebExtensions, so please check them out!

AMO technical architecture

addons.mozilla.org (AMO) has been around for more than 12 years, making it one of the oldest websites at Mozilla. It celebrated its 10th anniversary a couple of years ago, as Wil blogged about.

AMO started as a PHP site that grew and grew as new pieces of functionality were bolted on. In October 2009 the rewrite from PHP to Python began. New features were added, the site grew ever larger, and now a few cracks are starting to appear. These are merely the result of a site that has lots of features and functionality and has been around for a long time.

The site architecture is currently something like below, but please note this simplifies the site and ignores the complexities of AWS, the CDN and other parts of the site.image01

Basically, all the code is one repository and the main application (a Django app) is responsible for generating everything—from HTML, to emails, to APIs, and it all gets deployed at the same time. There’s a few problems with this:

  • The amount of functionality in the site has caused such a growth in interactions between the features that it is harder and harder to test.
  • Large JavaScript parts of the site have no automated testing.
  • The JavaScript and CSS spill over between different parts of the site, so changes in one regularly break other parts of the site.
  • Not all parts of the site have the same expectation of uptime but are all deployed at the same time.
  • Not all parts of the site have the same requirements for code contributions.

We are moving towards a new model similar to the one used for Firefox Marketplace. Whereas Marketplace built its own front-end framework, we are going to be using React on the front end.

The end result will start look something like this:

image00

A separate version of the site is rendered for the different use cases, for example developers or users. In this case a request comes in hits one of the appropriate front-end stacks. That will render the site using React universal in node.js on the server. It will access the data store by calling the appropriate Python REST APIs.

In this scenario, the legacy Python code will migrate to being a REST API that manages storage, transactions, workflow, permissions and the like. All the front-facing user interface work will be done in React and be independent from each other as much as possible.

It’s not quite micro services, but the breaking of a larger site into smaller independent pieces. The first part of this is happening with the “discovery pane” (accessible at about:addons). This is our first project using this infrastructure, which features a new streamlined way to install add-ons with a new technical architecture to serve it to users.

As we roll out this new architecture we’ll be doing more blog posts, so if you’d like to get involved then join our mailing list or check out our repositories on Github.