Migrating Popup ALT Attribute from XUL/XPCOM to WebExtensions

Firefox

Today’s post comes from Piro, the developer of Popup ALT Attribute, in addition to 40 other add-ons. He shares his thoughts about migrating XUL/XPCOM add-ons to WebExtensions, and shows us how he did it with Popup ALT Attribute. You can see the full text of this post on his personal blog.

***

Hello, add-on developers. My name is YUKI Hiroshi aka Piro, a developer of Firefox add-ons. For many years I developed Firefox and Thunderbird add-ons personally and for business, based on XUL and XPCOM.

I recently started to research the APIs are required to migrate my add-ons to WebExtensions, because Mozilla announced that XUL/XPCOM add-ons will be deprecated at the end of 2017. I realized that only some add-ons can be migrated with currently available APIs, and
Popup ALT Attribute is one such add-on.

Here is the story of how I migrated it.

What’s the add-on?

Popup ALT Attribute is an ancient add-on started in 2002, to show what is written in the alt attribute of img HTML elements on web pages. By default, Firefox shows only the title attribute as a tooltip.

Initially, the add-on was implemented to replace an internal function FillInHTMLTooltip() of Firefox itself.

In February 2016, I migrated it to be e10s-compatible. It is worth noting that depending on your add-on, if you can migrate it directly to WebExtensions, it will be e10s-compatible by default.

Re-formatting in the WebExtensions style

I read the tutorial on how to build a new simple WebExtensions-based add-on from scratch before migration, and I realized that bootstrapped extensions are similar to WebExtensions add-ons:

  • They are dynamically installed and uninstalled.
  • They are mainly based on JavaScript code and some static manifest files.

My add-on was easily re-formatted as a WebExtensions add-on, because I already migrated it to bootstrapped.

This is the initial version of the manifest.json I wrote. There were no localization and options UI:

{
  "manifest_version": 2,
  "name": "Popup ALT Attribute",
  "version": "4.0a1",
  "description": "Popups alternate texts of images or others like NetscapeCommunicator(Navigator) 4.x, and show long descriptions in the multi-row tooltip.",
  "icons": { "32": "icons/icon.png" },
  "applications": {
    "gecko": { "id": "{61FD08D8-A2CB-46c0-B36D-3F531AC53C12}",
               "strict_min_version": "48.0a1" }
  },
  "content_scripts": [
    { "all_frames": true,
      "matches": [""],
      "js": ["content_scripts/content.js"],
      "run_at": "document_start" }
  ]
}

I had already separated the main script to a frame script and a loader for it. On the other hand, manifest.json can have some manifest keys to describe how scripts are loaded. It means that I don’t need to put my custom loaders in the package anymore. Actually, a script for any web page can be loaded with the content_scripts rule in the above sample. See the documentation for content_scripts for more details.

So finally only 3 files were left.

Before:

+ install.rdf
+ icon.png
+ [components]
+ [modules]
+ [content]
    + content-utils.js

And after:

+ manifest.json (migrated from install.rdf)
+ [icons]
|   + icon.png (moved)
+ [content_scripts]
    + content.js (moved and migrated from content-utils.js)

And I still had to isolate my frame script from XPCOM.

  • The script touched nsIPrefBranch and some XPCOM components via XPConnect, so they were temporarily commented out.
  • User preferences were not available and only default configurations were there as fixed values.
  • Some constant properties accessed, like Ci.nsIDOMNode.ELEMENT_NODE, had to be replaced as Node.ELEMENT_NODE.
  • The listener for mousemove events from web pages was attached to the global namespace for a frame script, but it was re-attached to the document itself of each web page, because the script was now executed on each web page directly.

Localization

For the old install.rdf I had a localized description. In WebExtensions add-ons I had to do it in different way. See how to localize messages for details. In short I did the following:

Added files to define localized descriptions:

+ manifest.json
+ [icons]
+ [content_scripts]
+ [_locales]
    + [en_US]
    |   + messages.json (added)
    + [ja]
        + messages.json (added)

Note, en_US is different from en-US in install.rdf.

English locale, _locales/en_US/messages.json was:

{
  "name": { "message": "Popup ALT Attribute" },
  "description": { "message": "Popups alternate texts of images or others like NetscapeCommunicator(Navigator) 4.x, and show long descriptions in the multi-row tooltip." }
}

Japanese locale, _locales/ja/messages.json was also included. And, I had to update my manifest.json to embed localized messages:

{
  "manifest_version": 2,
  "name": "__MSG_name__",
  "version": "4.0a1",
  "description": "__MSG_description__",
  "default_locale": "en_US",
  ...

__MSG_****__ in string values are automatically replaced to localized messages. You need to specify the default locale manually via the default_locale key.

Sadly, Firefox 45 does not support the localization feature, so you need to use Nightly 48.0a1 or newer to try localization.

User preferences

Currently, WebExtensions does not provide any feature completely compatible to nsIPrefBranch. Instead, there are simple storage APIs. It can be used like an alternative of nsIPrefBranch to set/get user preferences. This add-on had no configuration UI but had some secret preferences to control its advanced features, so I did it for future migrations of my other add-ons, as a trial.

Then I encountered a large limitation: the storage API is not available in content scripts. I had to create a background script just to access the storage, and communicate with it via the inter-sandboxes messaging system. [Updated 4/27/16: bug 1197346 has been fixed on Nightly 49.0a1, so now you don’t need any hack to access the storage system from content scripts anymore. Now, my library (Configs.js) just provides easy access for configuration values instead of the native storage API.]

Finally, I created a tiny library to do that. I don’t describe how I did it here, but if you hope to know details, please see the source. There are just 177 lines.

I had to update my manifest.json to use the library from both the background page and the content script, like:

  "background": {
    "scripts": [
      "common/Configs.js", /* the library itself */
      "common/common.js"   /* codes to use the library */
    ]
  },
  "content_scripts": [
    { "all_frames": true,
      "matches": [""],
      "js": [
        "common/Configs.js", /* the library itself */
        "common/common.js",  /* codes to use the library */
        "content_scripts/content.js"
      ],
      "run_at": "document_start" }
  ]

Scripts listed in the same section share a namespace for the section. I didn’t have to write any code like require() to load a script from others. Instead, I had to be careful about the listing order of scripts, and wrote a script requiring a library after the library itself, in each list.

One last problem was: how to do something like the about:config or the MCD — general methods to control secret preferences across add-ons.

For my business clients, I usually provide add-ons and use MCD to lock their configurations. (There are some common requirements for business use of Firefox, so combinations of add-ons and MCD are more reasonable than creating private builds of Firefox with different configurations for each client.)

I think I still have to research around this point.

Options UI

WebExtensions provides a feature to create options pages for add-ons. It is also not supported on Firefox 45, so you need to use Nightly 48.0a1 for now. As I previously said, this add-on didn’t have its configuration UI, but I implemented it as a trial.

In XUL/XPCOM add-ons, rich UI elements like , , , and more are available, but these are going away at the end of next year. So I had to implement a custom configuration UI based on pure HTML and JavaScript. (If you need more rich UI elements, some known libraries for web applications will help you.)

On this step I created two libraries:

Conclusion

I’ve successfully migrated my Popup ALT Attribute add-on from XUL/XPCOM to WebExtensions. Now it is just a branch but I’ll release it after Firefox 48 is available.

Here are reasons why I could do it:

  • It was a bootstrapped add-on, so I had already isolated the add-on from all destructive changes.
  • The core implementation of the add-on was similar to a simple user script. Essential actions of the add-on were enclosed inside the content area, and no privilege was required to do that.

However, it is a rare case for me. My other 40+ add-ons require some privilege, and/or they work outside the content area. Most of my cases are such non-typical add-ons.

I have to do triage, plan, and request new APIs not only for me but for other XUL/XPCOM add-on developers also.

Thank you for reading.

Add-ons Update – Week of 2016/04/20

Firefox

I post these updates every 3 weeks to inform add-on developers about the status of the review queues, add-on compatibility, and other happenings in the add-ons world.

The Review Queues

In the past 3 weeks, 1226 listed add-ons were reviewed:

  • 1160 (95%) were reviewed in fewer than 5 days.
  • 45 (4%) were reviewed between 5 and 10 days.
  • 21 (1%) were reviewed after more than 10 days.

There are 73 listed add-ons awaiting review.

You can read about the recent improvements in the review queues here.

If you’re an add-on developer and are looking for contribution opportunities, please consider joining us. Add-on reviewers get invited to Mozilla events and earn cool gear with their work. Visit our wiki page for more information.

Compatibility Communications

Most of you should have received an email from us about the future compatibility of your add-ons. You can use the compatibility tool to enter your add-on ID and get some info on what we think is the best path forward for your add-on.

To ensure long-term compatibility, we suggest you start looking into WebExtensions, or use the Add-ons SDK and try to stick to the high-level APIs. There are many XUL add-ons that require APIs that aren’t available in either of these options, which is why we’re also asking you to fill out this survey, so we know which APIs we should look into adding to WebExtensions.

We’re holding regular office hours for Multiprocess Firefox compatibility, to help you work on your add-ons, so please drop in on Tuesdays and chat with us!

Firefox 47 Compatibility

The compatibility blog post for 47 is up. The bulk validation will be run soon. Make sure that the compatibility metadata for your add-on is up to date, so you don’t miss these checks.

As always, we recommend that you test your add-ons on Beta and Firefox Developer Edition to make sure that they continue to work correctly. End users can install the Add-on Compatibility Reporter to identify and report any add-ons that aren’t working anymore.

Extension Signing

The wiki page on Extension Signing has information about the timeline, as well as responses to some frequently asked questions. The current plan is to remove the signing override preference in Firefox 47 (updated from 46).

Developing Extensions With Web-ext 1.0

Firefox

As the transition to WebExtensions continues, we are also building tools to make developing them faster and easier than ever. Our latest is a command line tool called web-ext, which we released recently as an initial working version with some basic features.

We built it because we want developing extensions to be fast and easy. Just as Firefox’s WebExtensions API is designed for cross-browser compatibility, we want web-ext to eventually support platforms such as Chrome or Opera. We will continue developing jpm in parallel, as needed.

To give it a try, you can install it from npm:

npm install --global web-ext

When developing an extension, here’s how you can run it in Firefox to test it out:

cd /path/to/your/source
web-ext run

This is similar to how you can load your source directly on the about:debugging page.

When you’ve got your extension working, here’s how to build an XPI file that can be submitted to addons.mozilla.org:

web-ext build

You can also self-host your XPI file for distribution but it needs to be signed by Mozilla first. Here’s how to build and sign an XPI file:

web-ext sign

The end user documentation is a work in progress but you can reference all commands and options by typing:

web-ext --help

As you can see, this is a very early release just to get you started while we continue adding features. If you’d like to help out on the development of web-ext, check out the contributor guide and take a look at some good first bugs.

The “Why” of Electrolysis

Firefox

A multi-process architecture is finally coming to Firefox. Known by its codename, “Electrolysis” or “e10s,” this project aims to split the Firefox browser into a single process for the UI, and several processes for web content, media playback, plugins, etc.

Electrolysis represents a long overdue modernization of Firefox’s codebase: Internet Explorer has been multi-process since IE 8 Beta 1 in early 2008, Chrome followed six months later, and even Safari went multi-process in the summer of 2011. Put another way, Firefox has been the only major browser using a single process architecture for the past five years.

Benefits for Users

Though the result should be visually indistinguishable from a single-process Firefox, the multi-process model offers many compelling advantages in responsiveness, stability, performance, and security:

  • Responsiveness. By placing web content into a separate process from the browser interface, we can ensure that Firefox’s UI stays snappy even when a page is doing heavy computation or undergoing garbage collection.
  • Stability. Tabs shouldn’t crash, but when they do, isolating web content in separate processes means they can crash without taking down the rest of the browser.
  • Performance. A multi-process architecture lets us spread work across the many CPU cores present in modern computers and smartphones, resulting in more efficient utilization of the underlying hardware.
  • Security. Process separation makes it easier to implement restrictive security sandboxes for web content.

A common concern is that switching to a multi-processes architecture will dramatically increase Firefox’s memory usage. This is not the case. While multiple processes will have a greater memory footprint than a single process, the impact should be limited: we’re currently seeing multi-process Firefox use 10-20% more memory, however, it still uses half the memory of Chrome with the same sites loaded.

To ensure we don’t consume too much RAM, the first release of e10s will only use a single additional process for web content. We’ll add more processes in subsequent releases, as we become more memory efficient.

Challenges for Add-on Developers

The move to multi-process is an investment in the future: we’re paying down technical debt and redesigning Firefox’s architecture at a fundamental level. Like any change of this magnitude, there are associated challenges:

1. Cross-Process Communication. Before e10s, add-ons were accustomed to having direct access to both the browser context and web page content. With e10s, those two contexts are now two separate processes which must communicate through asynchronous message passing.

Many, but not all, add-ons will require modification to work efficiently with this new design. Add-ons that do not access web content, only use high-level SDK APIs, or are written with the new WebExtension APIs will not need modification.

You can see the compatibility status of popular add-ons at AreWeE10SYet.com, or use the compatibility checker to see what changes your add-on might need.

2. Performance. To help with compatibility during the transition to multi-process, we’ve implemented several temporary shims that allow backwards-compatible, synchronous communication between the browser and web content. This lets add-ons work as if Firefox was still single-process, but with potentially worse performance than before, since these shims must completely block multiple processes and incur overhead for inter-process communication.

The shims are available to add-ons by default, but authors should always prefer asynchronous message passing, high-level SDK APIs, or WebExtensions instead of shims.

You can disable shims for your add-on by setting the multiprocess permission to true in your package.json, or set to true in your install.rdf. This is how we know your add-on is e10s-compatible and will ensure that it won’t be marked as incompatible when e10s ships with Firefox.

Try it Today

Multi-process has been enabled by default in Firefox Developer Edition since version 42, released last August. If you’re an add-on developer or Firefox enthusiast, we highly recommend you try it out and report any bugs you encounter, either in Bugzilla or in the comments below. The Mozilla Developer Network also hosts extensive documentation on multi-process Firefox, including migration guides for add-on authors. Check it out, and let us know what you think!

Add-on Compatibility for Firefox 47

Firefox

Firefox 47 will be released on June 7th. Here’s the list of changes that went into this version that can affect add-on compatibility. There is more information available in Firefox 47 for Developers, so you should also give it a look.

General

Tabs

XPCOM

New

Let me know in the comments if there’s anything missing or incorrect on these lists. If your add-on breaks on Firefox 47, I’d like to know.

The automatic compatibility validation and upgrade for add-ons on AMO will happen in a few weeks, so keep an eye on your email if you have an add-on listed on our site with its compatibility set to Firefox 46.

Improving Review Time by Providing Links to Third Party Sources

Firefox

Earlier I wrote an article about submitting add-ons with sources attached, which is primarily needed for obfuscated or minified add-ons. I only briefly mentioned the specific cases where attaching sources is in fact NOT recommended. If you’ve determined you do not need to upload sources but you still want to profit from blazing fast review speeds, please read on.

To reiterate, you do not need to upload source code if your add-on only contains (minified) third-party libraries, or if the libraries you are calling via js-ctypes are system libraries or open source. What you need to do instead is provide links to the exact file versions in the repositories of the respective libraries. You should add these links in the “Notes to Reviewers” section when viewing or uploading your new version.

Just as with your add-on code, we review third-party library code for potential issues. When we notice a library is used more often, we can add the (sha256) checksum to a list of known libraries that are specially marked in the editor tools. Reviewers do not need to review this library again, which saves a lot of time.

The downside to using checksums is that the file must match exactly, down to the byte. A common issue we encounter is libraries being retrieved from non-official sites, for example CDNs. These sites may make slight changes, often something simple like white spaces, or adding/removing a source map URL. Similarly, some developers copy/paste the libraries into a text editor, which may result in different line endings.

Now to the important part: how to specify the repository links. You don’t need to do this for the common libraries mentioned in our list (currently: angular, backbone, backbone.localStorage, crypto.js, dojo, jquery, jqueryui, moment, mootools, prototype.js, scriptaculous, swfobject.js, underscore, yui). For all other libraries, we need the link to an exact copy of the file you have submitted, from the official website or repository. Here is an example:

Let’s say you are using the minified version of mousetrap version 1.4.2 because you haven’t had the chance to update to the latest version.

If the library is on github, you can usually find this version under the “releases” link, then clicking on the small tag icon next to the version number, then navigating to the file in the repository. For bonus points, if you are using the minifed version, also provide a link to the unminifed version beside it. If the library does not use github releases but instead just gives you a zip to download, provide the link to the zip instead.

Mentioning the links upfront will shorten the review time since we can check the library right away instead of asking you about it, making reviews faster. Leave a comment if you have any questions!