Migrating to WebExtensions: port your stored data

Firefox

WebExtensions are the new standard for add-on development in Firefox, and will be the only supported type of extension in release versions of Firefox later this year. Starting in Firefox 57, which is scheduled to arrive in November 2017, extensions other than WebExtensions will not load, and developers should be preparing to migrate their legacy extensions to WebExtensions.

If you have a legacy extension that writes data to the filesystem, and you’re planning to port it to WebExtensions, Embedded WebExtensions are available now in Firefox 51 to help you transition. Embedded WebExtensions can be used to transfer the stored data of your add-on to a format that can be used by WebExtensions. This is essential because it lets you to convert your users without the need for them to take any actions.

What is an Embedded WebExtension?

An Embedded WebExtension is an extension that combines two types of extensions in one, by incorporating a WebExtension inside of a bootstrapped or SDK extension.

Why use an Embedded WebExtension?

There are attributes (functions) of legacy add-ons that are used to store information related to the add-on that are not available in WebExtensions. Examples of these functions include user preferences, arbitrary file system access for storing assets, configuration information, stateful information, and others. If your add-on makes use of functionality like these to store information, you can use an Embedded WebExtension to access your legacy add-on data and move it over to a WebExtension. The earlier you do this, the more likely all your users will transition over smoothly.

It’s important to emphasize that Embedded WebExtensions are intended to be a transition tool, and will not be supported past Firefox 57. They should not be used for add-ons that are not expected to transition to WebExtensions.

How do I define an Embedded WebExtension?

To get started, read the documentation below. You can also contact us—we’re here to help you through the transition.

MDN docs: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Embedded_WebExtensions

Examples: https://github.com/mdn/webextensions-examples/tree/master/embedded-webextension-bootstrapped

https://github.com/mdn/webextensions-examples/tree/master/embedded-webextension-sdk

How Video DownloadHelper Became Compatible with Multiprocess Firefox

Firefox

Today’s post comes from Michel Gutierrez (mig), the developer of Video DownloadHelper, among other add-ons. He shares his story about the process of modernizing his XUL add-on to make it compatible with multiprocess Firefox (e10s).

***

Video DownloadHelper (VDH) is an add-on that extracts videos and image files from the Internet and saves them to your hard drive. As you surf the Web, VDH will show you a menu of download options when it detects something it can save for you.

It was first released in July 2006, when Firefox was on version 1.5. At the time, both the main add-on code and DOM window content were running in the same process. This was helpful because video URLs could easily be extracted from the window content by the add-on. The Smart Naming feature was also able to extract video names from the Web page.

When multiprocess Firefox architecture was first discussed, it was immediately clear that VDH needed a full rewrite with a brand new architecture. In multiprocess Firefox, DOM content for webpages run in a separate process, which means required asynchronous communication with the add-on code would increase significantly. It wasn’t possible to simply make adaptations to the existing code and architecture because it would make the code hard to read and unmaintainable.

The Migration

After some consideration, we decided to update the add-on using SDK APIs. Here were our requirements:

  • Code running in the content process needed to run separately from code running in Javascript modules and the main process. Communication must occur via message passing.
  • Preferences needed to be available in the content process, as there are many adjustable parameters that affect the user interface.
  • Localization of HTML pages within the content script should be as easy as possible.

In VDH, the choice was made to handle all of these requirements using the same Client-Server architecture commonly used in regular Web applications: the components that have access to the preferences, localization, and data storage APIs (running in the main process) serve this data to the UI components and the components injected into the page (running in the content process), through the messaging API provided by the SDK.

Limitations

Migrating to the SDK enabled us to become compatible with multiprocess Firefox, but it wasn’t a perfect solution. Low-level SDK APIs, which aren’t guaranteed to work with e10s or stay compatible with future versions of Firefox, were required to implement anything more than simple features. Also, an increased amount of communication between processes is required even for seemingly simple interactions.

  • Resizing content panels can only occur in the background process, but only the content process knows what the dimensions should be. This gets more complicated when the size dynamically changes or depends on various parameters.
  • Critical features like monitoring network traffic or launching external programs in VDH requires low-level APIs.
  • Capturing tab thumbnails from the Add-on SDK API does not work in e10s mode. This feature had to be reimplemented in the add-on using a framescript.
  • When intercepting network responses, the Add-on SDK does not decode compressed responses.
  • The SDK provides no easy means to determine if e10s is enabled or not, which would be useful as long as glitches remain where the add-on has to act differently.

Future Direction

Regardless of the limitations posed, making VDH compatible to multiprocess Firefox was a great success. Taking the time to rewrite the add-on also improved the general architecture and prepared it for changes needed for WebExtensions. The first e10s-compatible version of VDH is version 5.0.1 and had been available since March 2015.

Looking forward, the next big challenge is making VDH compatible with WebExtensions. We considered migrating directly to WebExtensions, but the legacy and low-level SDK APIs used in VDH could not be replaced at the time without compromising the add-on’s features.

To fully complete the transition to WebExtensions, additional APIs may need to be created. As an extension developer we’ve found it helpful to work with Mozilla to define those APIs, and design them in a way that is general enough for them to be useful in many other types of add-ons.

A note from the add-ons team: resources for migrating your add-ons to WebExtensions can be found here.

Migrating Popup ALT Attribute from XUL/XPCOM to WebExtensions

Firefox

Today’s post comes from Piro, the developer of Popup ALT Attribute, in addition to 40 other add-ons. He shares his thoughts about migrating XUL/XPCOM add-ons to WebExtensions, and shows us how he did it with Popup ALT Attribute. You can see the full text of this post on his personal blog.

***

Hello, add-on developers. My name is YUKI Hiroshi aka Piro, a developer of Firefox add-ons. For many years I developed Firefox and Thunderbird add-ons personally and for business, based on XUL and XPCOM.

I recently started to research the APIs are required to migrate my add-ons to WebExtensions, because Mozilla announced that XUL/XPCOM add-ons will be deprecated at the end of 2017. I realized that only some add-ons can be migrated with currently available APIs, and
Popup ALT Attribute is one such add-on.

Here is the story of how I migrated it.

What’s the add-on?

Popup ALT Attribute is an ancient add-on started in 2002, to show what is written in the alt attribute of img HTML elements on web pages. By default, Firefox shows only the title attribute as a tooltip.

Initially, the add-on was implemented to replace an internal function FillInHTMLTooltip() of Firefox itself.

In February 2016, I migrated it to be e10s-compatible. It is worth noting that depending on your add-on, if you can migrate it directly to WebExtensions, it will be e10s-compatible by default.

Re-formatting in the WebExtensions style

I read the tutorial on how to build a new simple WebExtensions-based add-on from scratch before migration, and I realized that bootstrapped extensions are similar to WebExtensions add-ons:

  • They are dynamically installed and uninstalled.
  • They are mainly based on JavaScript code and some static manifest files.

My add-on was easily re-formatted as a WebExtensions add-on, because I already migrated it to bootstrapped.

This is the initial version of the manifest.json I wrote. There were no localization and options UI:

{
  "manifest_version": 2,
  "name": "Popup ALT Attribute",
  "version": "4.0a1",
  "description": "Popups alternate texts of images or others like NetscapeCommunicator(Navigator) 4.x, and show long descriptions in the multi-row tooltip.",
  "icons": { "32": "icons/icon.png" },
  "applications": {
    "gecko": { "id": "{61FD08D8-A2CB-46c0-B36D-3F531AC53C12}",
               "strict_min_version": "48.0a1" }
  },
  "content_scripts": [
    { "all_frames": true,
      "matches": [""],
      "js": ["content_scripts/content.js"],
      "run_at": "document_start" }
  ]
}

I had already separated the main script to a frame script and a loader for it. On the other hand, manifest.json can have some manifest keys to describe how scripts are loaded. It means that I don’t need to put my custom loaders in the package anymore. Actually, a script for any web page can be loaded with the content_scripts rule in the above sample. See the documentation for content_scripts for more details.

So finally only 3 files were left.

Before:

+ install.rdf
+ icon.png
+ [components]
+ [modules]
+ [content]
    + content-utils.js

And after:

+ manifest.json (migrated from install.rdf)
+ [icons]
|   + icon.png (moved)
+ [content_scripts]
    + content.js (moved and migrated from content-utils.js)

And I still had to isolate my frame script from XPCOM.

  • The script touched nsIPrefBranch and some XPCOM components via XPConnect, so they were temporarily commented out.
  • User preferences were not available and only default configurations were there as fixed values.
  • Some constant properties accessed, like Ci.nsIDOMNode.ELEMENT_NODE, had to be replaced as Node.ELEMENT_NODE.
  • The listener for mousemove events from web pages was attached to the global namespace for a frame script, but it was re-attached to the document itself of each web page, because the script was now executed on each web page directly.

Localization

For the old install.rdf I had a localized description. In WebExtensions add-ons I had to do it in different way. See how to localize messages for details. In short I did the following:

Added files to define localized descriptions:

+ manifest.json
+ [icons]
+ [content_scripts]
+ [_locales]
    + [en_US]
    |   + messages.json (added)
    + [ja]
        + messages.json (added)

Note, en_US is different from en-US in install.rdf.

English locale, _locales/en_US/messages.json was:

{
  "name": { "message": "Popup ALT Attribute" },
  "description": { "message": "Popups alternate texts of images or others like NetscapeCommunicator(Navigator) 4.x, and show long descriptions in the multi-row tooltip." }
}

Japanese locale, _locales/ja/messages.json was also included. And, I had to update my manifest.json to embed localized messages:

{
  "manifest_version": 2,
  "name": "__MSG_name__",
  "version": "4.0a1",
  "description": "__MSG_description__",
  "default_locale": "en_US",
  ...

__MSG_****__ in string values are automatically replaced to localized messages. You need to specify the default locale manually via the default_locale key.

Sadly, Firefox 45 does not support the localization feature, so you need to use Nightly 48.0a1 or newer to try localization.

User preferences

Currently, WebExtensions does not provide any feature completely compatible to nsIPrefBranch. Instead, there are simple storage APIs. It can be used like an alternative of nsIPrefBranch to set/get user preferences. This add-on had no configuration UI but had some secret preferences to control its advanced features, so I did it for future migrations of my other add-ons, as a trial.

Then I encountered a large limitation: the storage API is not available in content scripts. I had to create a background script just to access the storage, and communicate with it via the inter-sandboxes messaging system. [Updated 4/27/16: bug 1197346 has been fixed on Nightly 49.0a1, so now you don’t need any hack to access the storage system from content scripts anymore. Now, my library (Configs.js) just provides easy access for configuration values instead of the native storage API.]

Finally, I created a tiny library to do that. I don’t describe how I did it here, but if you hope to know details, please see the source. There are just 177 lines.

I had to update my manifest.json to use the library from both the background page and the content script, like:

  "background": {
    "scripts": [
      "common/Configs.js", /* the library itself */
      "common/common.js"   /* codes to use the library */
    ]
  },
  "content_scripts": [
    { "all_frames": true,
      "matches": [""],
      "js": [
        "common/Configs.js", /* the library itself */
        "common/common.js",  /* codes to use the library */
        "content_scripts/content.js"
      ],
      "run_at": "document_start" }
  ]

Scripts listed in the same section share a namespace for the section. I didn’t have to write any code like require() to load a script from others. Instead, I had to be careful about the listing order of scripts, and wrote a script requiring a library after the library itself, in each list.

One last problem was: how to do something like the about:config or the MCD — general methods to control secret preferences across add-ons.

For my business clients, I usually provide add-ons and use MCD to lock their configurations. (There are some common requirements for business use of Firefox, so combinations of add-ons and MCD are more reasonable than creating private builds of Firefox with different configurations for each client.)

I think I still have to research around this point.

Options UI

WebExtensions provides a feature to create options pages for add-ons. It is also not supported on Firefox 45, so you need to use Nightly 48.0a1 for now. As I previously said, this add-on didn’t have its configuration UI, but I implemented it as a trial.

In XUL/XPCOM add-ons, rich UI elements like , , , and more are available, but these are going away at the end of next year. So I had to implement a custom configuration UI based on pure HTML and JavaScript. (If you need more rich UI elements, some known libraries for web applications will help you.)

On this step I created two libraries:

Conclusion

I’ve successfully migrated my Popup ALT Attribute add-on from XUL/XPCOM to WebExtensions. Now it is just a branch but I’ll release it after Firefox 48 is available.

Here are reasons why I could do it:

  • It was a bootstrapped add-on, so I had already isolated the add-on from all destructive changes.
  • The core implementation of the add-on was similar to a simple user script. Essential actions of the add-on were enclosed inside the content area, and no privilege was required to do that.

However, it is a rare case for me. My other 40+ add-ons require some privilege, and/or they work outside the content area. Most of my cases are such non-typical add-ons.

I have to do triage, plan, and request new APIs not only for me but for other XUL/XPCOM add-on developers also.

Thank you for reading.

The “Why” of Electrolysis

Firefox

A multi-process architecture is finally coming to Firefox. Known by its codename, “Electrolysis” or “e10s,” this project aims to split the Firefox browser into a single process for the UI, and several processes for web content, media playback, plugins, etc.

Electrolysis represents a long overdue modernization of Firefox’s codebase: Internet Explorer has been multi-process since IE 8 Beta 1 in early 2008, Chrome followed six months later, and even Safari went multi-process in the summer of 2011. Put another way, Firefox has been the only major browser using a single process architecture for the past five years.

Benefits for Users

Though the result should be visually indistinguishable from a single-process Firefox, the multi-process model offers many compelling advantages in responsiveness, stability, performance, and security:

  • Responsiveness. By placing web content into a separate process from the browser interface, we can ensure that Firefox’s UI stays snappy even when a page is doing heavy computation or undergoing garbage collection.
  • Stability. Tabs shouldn’t crash, but when they do, isolating web content in separate processes means they can crash without taking down the rest of the browser.
  • Performance. A multi-process architecture lets us spread work across the many CPU cores present in modern computers and smartphones, resulting in more efficient utilization of the underlying hardware.
  • Security. Process separation makes it easier to implement restrictive security sandboxes for web content.

A common concern is that switching to a multi-processes architecture will dramatically increase Firefox’s memory usage. This is not the case. While multiple processes will have a greater memory footprint than a single process, the impact should be limited: we’re currently seeing multi-process Firefox use 10-20% more memory, however, it still uses half the memory of Chrome with the same sites loaded.

To ensure we don’t consume too much RAM, the first release of e10s will only use a single additional process for web content. We’ll add more processes in subsequent releases, as we become more memory efficient.

Challenges for Add-on Developers

The move to multi-process is an investment in the future: we’re paying down technical debt and redesigning Firefox’s architecture at a fundamental level. Like any change of this magnitude, there are associated challenges:

1. Cross-Process Communication. Before e10s, add-ons were accustomed to having direct access to both the browser context and web page content. With e10s, those two contexts are now two separate processes which must communicate through asynchronous message passing.

Many, but not all, add-ons will require modification to work efficiently with this new design. Add-ons that do not access web content, only use high-level SDK APIs, or are written with the new WebExtension APIs will not need modification.

You can see the compatibility status of popular add-ons at AreWeE10SYet.com, or use the compatibility checker to see what changes your add-on might need.

2. Performance. To help with compatibility during the transition to multi-process, we’ve implemented several temporary shims that allow backwards-compatible, synchronous communication between the browser and web content. This lets add-ons work as if Firefox was still single-process, but with potentially worse performance than before, since these shims must completely block multiple processes and incur overhead for inter-process communication.

The shims are available to add-ons by default, but authors should always prefer asynchronous message passing, high-level SDK APIs, or WebExtensions instead of shims.

You can disable shims for your add-on by setting the multiprocess permission to true in your package.json, or set to true in your install.rdf. This is how we know your add-on is e10s-compatible and will ensure that it won’t be marked as incompatible when e10s ships with Firefox.

Try it Today

Multi-process has been enabled by default in Firefox Developer Edition since version 42, released last August. If you’re an add-on developer or Firefox enthusiast, we highly recommend you try it out and report any bugs you encounter, either in Bugzilla or in the comments below. The Mozilla Developer Network also hosts extensive documentation on multi-process Firefox, including migration guides for add-on authors. Check it out, and let us know what you think!