Automated Testing¶

You’ve just written a feature and (hopefully!) want to test it. Or you’ve decided that an existing feature doesn’t have enough tests and want to contribute some. But where do you start? You’ve looked around and found references to things like “xpcshell” or “web-platform-tests” or “talos”. What code, features or platforms do they all test? Where do their feature sets overlap? In short, where should your new tests go? This document is a starting point for those who want to start to learn about Mozilla’s automated testing tools and procedures. Below you’ll find a short summary of each framework we use, and some questions to help you pick the framework(s) you need for your purposes.

If you still have questions, ask on Matrix or on the relevant bug.

Firefox Production¶

These tests are found within the mozilla-central tree, along with the product code.

They are run when a changeset is pushed to mozilla-central, autoland, or try, with the results showing up on Treeherder. Not all tests will be run on every changeset; alogrithms are put in place to run the most likely failures, with all tests being run on a regular basis.

They can also be run on local builds. Note: Most of the mobile tests run on emulators, but some of the tests (notably, performance tests) run on hardware devices. We try to avoid running mobile tests on hardware devices unnecessarily. In Treeherder, tests with names that start with “hw-” run on hardware.

Linting¶

Lint tests help to ensure better quality, less error-prone code by analysing the code with a linter.

Linters¶
Treeherder Symbol	Name	Platforms	What is Tested
`ES`	ESLint	All	JavaScript is analyzed for correctness.
`ES-build`	eslint-build	All	Extended javascript analysis that uses build artifacts.
`mocha(EPM)`	ESLint-plugin-mozilla	Desktop	The ESLint plugin rules.
`f8`	flake8	All	Python analyzed for style and correctness.
`stylelint`	Stylelint	All	CSS is analyzed for correctness.
`W`	wpt lint	Desktop	web-platform-tests analyzed for style and manifest correctness
`WR(tidy)`	WebRender servo-tidy	Desktop	Code in gfx/wr is run through servo-tidy.
`A`	Spotless	Android	Java is analyzed for style and correctness.

Functional testing¶

Automated Test Suites¶
Treeherder Symbol	Name	Platform	Process	Environment		Privilege	What is Tested
				Shell	Browser Profile
`R(J)`	JS Reftest	Desktop	N/A	JSShell	N/A	N/A	The JavaScript engine’s implementation of the JavaScript language.
`R(C)`	Crashtest	All	Child	Content	Yes	Low	That pages load without crashing, asserting, or leaking.
`R(R)`	Reftest	All	Child	Content	Yes	Low	That pages are rendered (and thus also layed out) correctly.
`GTest`	GTest	All	N/A	Terminal	N/A	N/A	Code that is not exposed to JavaScript.
`X`	xpcshell	All	Parent, Allow	XPCShell	Allow	High	Low-level code exposed to JavaScript, such as XPCOM components.
`M(a11y)`	Accessibility (mochitest-a11y)	Desktop	Child	Content	Yes	?	Accessibility interfaces.
`M(1), M(2), M(...)`	Mochitest plain	All	Child	Content	Yes	Low, Allow	Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test.
`M(c1/c2/c3)`	Mochitest chrome	All	Child, Allow	Content	Yes	High	Code requiring UI or JavaScript interactions with privileged objects.
`M(bc)`	Mochitest browser-chrome	All	Parent, Allow	Browser	Yes	High	How the browser UI interacts with itself and with content.
`M(remote)`	Mochitest Remote Protocol	All	Parent, Allow	Browser	Yes	High	Firefox Remote Protocol (Implements parts of Chrome dev-tools protocol). Based on Mochitest browser-chrome.
`SM(...), SM(pkg)`	SpiderMonkey automation	Desktop	N/A	JSShell	N/A	Low	SpiderMonkey engine shell tests and JSAPI tests.
`W`	web-platform-tests	Desktop	Child	Content	Yes	Low	Standardized features exposed to ECMAScript in web content; tests are shared with other vendors.
`Wr`	web-platform-tests	All	Child	Content	Yes	Low	Layout and graphic correctness for standardized features; tests are shared with other vendors.
`Mn`	Marionette	Desktop	?	Content, Browser	?	High	Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes.
`Fxfn`	Firefox UI Tests	Desktop	?	Content, Browser	Yes	High	Integration tests with a focus on the user interface and localization.
`tt(c)`	telemetry-tests-client	Desktop	N/A	Content, Browser	Yes	High	Integration tests for the Firefox Telemetry client.
`TV`	Test Verification (test-verify)	All	Depends on test harness	?	?	?	Uses other test harnesses - mochitest, reftest, xpcshell - to perform extra testing on new/modified tests.
`TVw`	Test Verification for wpt (test-verify-wpt)	Desktop	Child	?	?	?	Uses wpt test harnesses to perform extra testing on new/modified web-platform tests.
`WR(wrench)`	WebRender standalone tests	All	N/A	Terminal	N/A	N/A	WebRender rust code (as a standalone module, with Gecko integration).

Note: there are preference-based variations of the previous testing suites. For example, mochitests on Treeherder can have gli, swr, spi, nofis, a11y-checks, spi-nw-1proc, and many others. Another example is GTest, which can use GTest-1proc. To learn more about these variations, you can mouse hover over these items to read a description of what these abbreviations mean.

Table key¶

Symbol

Abbreviation for the test suite used by Treeherder. The first letter generally indicates which of the test harnesses is used to execute the test. The letter in parentheses identifies the actual test suite.

Name

Common name used when referring to the test suite.

File type

When adding a new test, you will generally create a file of this type in the source tree and then declare it in a manifest or makefile.

Platform

Most test suites are supported only on a subset of the available plaforms and operating systems. Unless otherwise noted:

Desktop tests run on Windows, Mac OS X, and Linux.
Mobile tests run on Android emulators or remotely on Android devices.

Process

When Parent is indicated, the test file will always run in the parent process, even when the browser is running in Electrolysis (e10s) mode.
When Child is indicated, the test file will run in the child process when the browser is running in Electrolysis (e10s) mode.
The Allow label indicates that the test has access to mechanisms to run code in the other process.

Environment

The JSShell and XPCShell environments are limited JavaScript execution environments with no windows or user interface (note however that XPCShell tests on Android are run within a browser window.)
The Content indication means that the test is run inside a content page loaded by a browser window.
The Browser indication means that the test is loaded in the JavaScript context of a browser XUL window.
The Browser Profile column indicates whether a browser profile is loaded when the test starts. The Allow label means that the test can optionally load a profile using a special command.

Privilege

Indicates whether the tests normally run with low (content) or high (chrome) JavaScript privileges. The Allow label means that the test can optionally run code in a privileged environment using a special command.

Performance testing¶

There are many test harnesses used to test performance. For more information on the various performance harnesses, check out the perf docs.

So which should I use?¶

Generally, you should pick the lowest-level framework that you can. If you are testing JavaScript but don’t need a window, use XPCShell or even JSShell. If you’re testing page layout, try to use web-platform-test reftest. The advantage in lower level testing is that you don’t drag in a lot of other components that might have their own problems, so you can home in quickly on any bugs in what you are specifically testing.

Here’s a series of questions to ask about your work when you want to write some tests.

Is it low-level code?¶

If the functionality is exposed to JavaScript, and you don’t need a window, consider XPCShell. If not, you’ll probably have to use GTest, which can test pretty much anything. In general, this should be your last option for a new test, unless you have to test something that is not exposed to JavaScript.

Does it cause a crash?¶

If you’ve found pages that crash Firefox, add a crashtest to make sure future versions don’t experience this crash (assertion or leak) again. Note that this may lead to more tests once the core problem is found.

Is it a layout/graphics feature?¶

Reftest is your best bet, if possible.

Do you need to verify performance?¶

Use an appropriate performance test suite from this list.

Are you testing UI features?¶

Try one of the flavors of mochitest, or Marionette if the application also needs to be restarted, or tested with localized builds.

Are you testing Mobile/Android?¶

If you are testing GeckoView, you will need to need to use JUnit integration tests.

There are some specific features that Mochitest or Reftest can cover. Browser-chrome tests do not run on Android. If you want to test performance, Raptor will be a good choice.

Are you doing none of the above?¶

To get your tests running in continuous integration, try web-platform-tests, or Mochitest, or, if higher privileges are required, try Mochitest browser chrome tests.
For Desktop Firefox, or if you just want to see the future of Gecko testing, look into the on-going Marionette project.

Need to get more data out of your tests?¶

Most test jobs now expose an environment variable named $MOZ_UPLOAD_DIR. If this variable is set during automated test runs, you can drop additional files into this directory, and they will be uploaded to a web server when the test finishes. The URLs to retrieve the files will be output in the test log.

Passing $MOZ_RECORD_TEST=1 as an environment variable when running some tests (e.g. mochitests) on Linux platform will trigger a recording of the desktop with GNOME Screencast.

Need to set preferences for test-suites?¶

First ask yourself if these prefs need to be enabled for all tests or just a subset of tests (e.g to enable a feature).

Setting prefs that only apply to certain tests¶

If the answer is the latter, try to set the pref as local to the tests that need it as possible. Here are some options:

If the test runs in chrome scope (e.g mochitest chrome or browser-chrome), you can use Services.prefs to set the prefs in your test’s setup function. Be sure to reset the pref back to its original value during teardown!
Mochitest plain tests can use SpecialPowers to set prefs.
All variants of mochitest can set prefs in their manifests. For example, to set a pref for all tests in a manifest:
```
[DEFAULT]
prefs =
  my.awesome.pref=foo,
  my.other.awesome.pref=bar,
[test_foo.js]
[test_bar.js]
```
All variants of reftest can also set prefs in their manifests.
All variants of web-platform-tests can also set prefs in their manifests.

Setting prefs that apply to the entire suite¶

Most test suites define prefs in user.js files that live under testing/profiles. Each directory is a profile that contains a user.js file with a number of prefs defined in it. Test suites will then merge one or more of these basic profiles into their own profile at runtime. To see which profiles apply to which test suites, you can inspect testing/profiles/profiles.json. Profiles at the beginning of the list get overridden by profiles at the end of the list.

Because this system makes it hard to get an overall view of which profiles are set for any given test suite, a handy profile utility was created:

$ cd testing/profiles
$ ./profile -- --help
usage: profile [-h] {diff,sort,show,rm} ...
$ ./profile show mochitest          # prints all prefs that will be set in mochitest
$ ./profile diff mochitest reftest  # prints differences between the mochitest and reftest suites

Note: JS engine tests do not use testing/profiles yet, instead set prefs here.

Adding New Context to Skip Conditions¶

Often when standing up new test configurations, it’s necessary to add new keys that can be used in skip-if annotations.