Compatibility Test Suite

Compatibility Test Suite (CTS) is a free, commercial-grade test suite used by device manufacturers to help ensure that their devices are compatible, and to report test results for validations. CTS is intended to be integrated into the daily workflow (such as through a continuous build system) of the engineers building a device. CTS's intent is to reveal incompatibilities early on, and ensure that the software remains compatible throughout the development process.

CTS is available for download as a binary or as source in the Android Open Source Project (AOSP). CTS runs on a desktop machine and executes test cases directly on attached devices or an emulator.

CTS contains two major software components:

  • The Trade Federation test harness runs on your desktop machine and manages test execution. Trade Federation offers the ability to shard tests across multiple Devices Under Test (DUTs). You may also use the suite retry feature to retry the failures only rather than the full suites, greatly reducing re-run time.
  • The individual test cases are executed on the DUT. The test cases are written in Java as JUnit tests and packaged Android .apk files to run on the actual device target.

Compatibility Test Suite Verifier (CTS Verifier)

The Compatibility Test Suite Verifier (CTS Verifier) is a supplement to CTS available for download. The CTS Verifier provides tests for APIs and functions that cannot be tested on a stationary device without manual input (for example, audio quality and accelerometer) or manual positioning.

The CTS Verifier is a tool for manual testing and includes the following software components:

  • The CTS Verifier app that is executed on the DUT (and on a 2nd device for distance measurement requirements) and collects the results.

  • The executable(s) or script(s) that are executed on the desktop machine to provide data or additional control for some test cases in the CTS Verifier app.

Workflow

CTS flow

Figure 1. How to use CTS

This diagram summarizes CTS workflow. Please refer to the subpages of this section starting with Setup for detailed instructions.

Types of test cases

CTS includes the following types of test cases:

  • Unit tests test atomic units of code within the Android platform; e.g. a single class, such as java.util.HashMap.

  • Functional tests test a combination of APIs together in a higher-level use-case.

Unit test coverage

The unit test cases cover the following areas to ensure compatibility:

Area Description
Signature tests For each Android release, there are XML files describing the signatures of all public APIs contained in the release. The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file.
Platform API Tests Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK Class Index to ensure API correctness, including correct class, attribute and method signatures, correct method behavior, and negative tests to ensure expected behavior for incorrect parameter handling.
Dalvik Tests The tests focus on testing the Dalvik Executable Format.
Platform Data Model The CTS tests the core platform data model as exposed to application developers through content providers, as documented in the SDK android.provider package: contacts, browser, settings, etc.
Platform Intents The CTS tests the core platform intents, as documented in the SDK Available Intents.
Platform Permissions The CTS tests the core platform permissions, as documented in the SDK Available Permissions.
Platform Resources The CTS tests for correct handling of the core platform resource types, as documented in the SDK Available Resource Types. This includes tests for: simple values, drawables, nine-patch, animations, layouts, styles and themes, and loading alternate resources.