Compatibility Test Suite (CTS) Framework User Manual

Android 4.0 CTS r1 Open Handset Alliance

Google Confidential

Contents 1. Why be compatible?....................................................................... 3 2. How can I become compatible? .................................................... 4 2.1. Comply with Android Compatibility Definition document............ 4 2.2. Pass the Compatibility Test Suite (CTS).................................... 4 2.3. Submit report ............................................................................. 4 3. How does the CTS work?............................................................... 5 3.1. Workflow .................................................................................... 5 3.2. Types of test cases .................................................................... 6 3.3. Areas Covered ........................................................................... 6 4. Setting up and using the CTS........................................................ 8 4.1. Configuring the CTS .................................................................. 8 4.2. Setting up your device ............................................................... 0 4.3. Using the CTS............................................................................ 0 4.4. Selecting CTS Plans .................................................................. 9 5. Interpreting the Test Results ....................................................... 11 6. Release Notes ............................................................................... 13 6.1. General .................................................................................... 13 6.2. Known Framework issues ........................................................ 13 6.3. Known Test issues ..................................................................... 0 7. Appendix: CTS Console Command Reference.......................... 14

Google Confidential

1. Why be compatible?

1. Give your users the best possible experience with the applications they run. When a device is compatible with Android, users can choose from among many high-quality applications. Applications that take full advantage of Android's features are likely to perform best on compatible devices. 2. Make it easy for developers to write top-quality applications for your device. Developers want to streamline their applications for Android, and this is easiest for them when they are writing for a predictable platform. 3. Take advantage of the Android Market. Compatible handsets can give users access to the Android Market. Android compatibility is free, and it's easy.

Google Confidential

2. How can I become compatible?

2.1. Comply with Android Compatibility Definition document

To start, read the Android compatibility definition for the Android platform version that you want. This document enumerates the software and the hardware features in a compatible Android device. Except where noted, the features are all required for Android compliance. To learn more about Android compatibility definition in general, and to locate and download a particular definitions document, see the current Compatibility Definition. Archived versions of older Compatibility Definitions may be found on the Downloads page.

2.2. Pass the Compatibility Test Suite (CTS)

The Compatibility Test Suite (CTS) is a downloadable open-source testing harness that you can use in any way you like as you develop your handset; for example, you could use the CTS to do continuous self-testing during your development work. For more about the CTS and the compatibility report that it generates, see the Compatibility Test Suite page. For instructions on using the CTS, see the CTS User Guide.

2.3. Submit report

When you are ready to claim compatibility for your device, you can submit the CTSgenerated report to [email protected]. When you submit a CTS report, you can also request access to the Android Market. * This is an early preview of CTS. The compatibility site and the service to certify your compatibility reports are work in progress - we will update you when these are ready.

Google Confidential

3. How does the CTS work?

The CTS is an automated testing harness that includes two major software components: The CTS test harness runs on your desktop machine and manages test execution. Individual test cases are executed on attached mobile devices or on an emulator. The test cases are written in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.

3.1. Workflow
1. Use the bundled CTS release or download the CTS from the Android Open Source Project onto your desktop machine. 2. Install and configure the CTS. 3. Attach at least one device (or emulator) to your machine. 4. Launch the CTS. The CTS test harness loads the test plan onto the attached devices. For each test in the test harness: The test harness pushes a .apk file to each device, executes the test through instrumentation, and records test results. The test harness removes the .apk file from each device. 5. Once all the tests are executed, you can view the test results in your browser and use the results to adjust your design. You can continue to run the CTS throughout your development process. When you are ready, you can submit the report generated by the CTS to

Google Confidential

[email protected]. The report is a .zip archived file that contains XML results and supplemental information such as screen captures.

3.2. Types of test cases

The CTS includes the following types of test cases: Unit tests test atomic units of code within the Android platform; e.g. a single class, such as java.util.HashMap. Functional tests test a combination of APIs together in a higher-level usecase. Reference application tests instrument a complete sample application to exercise a full set of APIs and Android runtime services Future versions of the CTS will include the following types of test cases: Robustness tests test the durability of the system under stress. Performance tests test the performance of the system against defined benchmarks, for example rendering frames per second.

3.3. Areas Covered

The unit test cases cover the following areas to ensure compatibility

Area

Description For each Android release, there are XML files describing the signatures of all public APIs contained in the release. The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file. Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK Class Index to ensure API correctness: correct class, attribute and method signatures correct method behavior negative tests to ensure expected behavior for incorrect parameter handling The tests focus on testing the Dalvik VM The CTS tests the core platform data model as exposed to application developers through content providers, as documented in the SDK android.provider package: contacts browser

Signature tests

Platform API Tests

Dalvik VM Tests

Platform Data Model

Google Confidential

 settings more... The CTS tests the core platform intents, as documented in the SDK Available Intents.

Platform Intents

The CTS tests the core platform permissions, as documented in the Platform Permissions SDK Available Permissions. The CTS tests for correct handling of the core platform resource types, as documented in the SDK Available Resource Types. This includes tests for: simple values drawables nine-patch animations layouts styles and themes loading alternate resources

Platform Resources

Google Confidential

4. Setting up and using the CTS

4.1. Configuring the CTS
Note: the steps to configure and run CTS have changed in the 4.0 release. To run CTS, make sure you have a recent copy of adb installed on your machine. To install adb, download and install the Android SDK tools. Then add the SDK Platform tools component. Ensure 'adb' is in your system path. Example: export PATH=$PATH:/home/myuser/android-sdk-linux_x86/ platform-tools

4.2. Setting up your device

CTS can be executed only on consumer devices. This section is important as not following these instructions will lead to test timeouts/ failures: 1. Your phone should be running a user build (Android 4.0 and later) from source.android.com 2. Please refer to this link on the Android developer site and set up your device accordingly. 3. Make sure that your device has been flashed with a user build (Android 4.0 and later) before you run CTS. 4. You need to ensure the Text To Speech files are installed on the device. You can check via Settings > Speech synthesis > Install voice data before running CTS tests. (Note that this assumes you have Android Market installed on the device, if not you will need to install the files manually via adb) 5. Make sure the device has a SD card plugged in and the card is empty. Warning: CTS may modify/erase data on the SD card plugged in to the device. 6. Do a factory data reset on the device (Settings > SD Card & phone storage > Factory data reset). Warning: This will erase all user data from the phone. 7. Make sure no lock pattern is set on the device (Settings > Security > Screen Lock should be 'None'). 8. Make sure the "USB Debugging" development option is checked (Settings > Developer options > USB debugging). 9. Make sure Settings > Developer options > Stay Awake is checked 10. Make sure Settings > Developer options > Allow mock locations is checked 11. Make sure device is connected to a functioning Wi-Fi network (Settings > WiFi)

Google Confidential

12. Make sure the device is at the home screen at the start of CTS (Press the home button). 13. While a device is running tests, it must not be used for any other tasks. 14. Do not press any keys on the device while CTS is running. Pressing keys or touching the screen of a test device will interfere with the running tests and may lead to test failures. 15. Set up accessibility tests: 1. adb install android-cts/repository/testcases/
CtsDelegatingAccessibilityService.apk

2. On the device, enable Settings > Accessibility > Delegating Accessibility Service 16. Set up device administration tests: 1. adb install android-cts/repository/testcases/
CtsDeviceAdmin.apk

2. On the device, enable Settings > Security > Device Administrators > android.deviceadmin.cts.CtsDeviceAdmin* settings

4.3. Using the CTS

To run a test plan: 1. Make sure you have at least one device connected. Launch the CTS console by running the cts-tradefed script e.g. $ ./android-cts/tools/cts-tradefed 2. You may start the default test plan (containing all of the test packages) by typing run cts --plan CTS. This will kick off all the CTS tests required for compatibility. Type list plans to see a list of test plans in the repository. Type list packages to see a list of test packages in the repository. See the CTS command reference or type help for a complete list of supported commands. 3. Alternately, you can just run a CTS plan from the command line using ctstradefed run cts --plan <plan_name> 4. You should test progress and results reported on the console.

4.4. Selecting CTS Plans

For this release the following 7 test plans are available. 1. CTS - contains all tests and will run ~17,000 tests on your device. These tests are required for compatibility. At this point performance tests are not part of this plan (this will change for future CTS releases). 2. Signature - contains the signature verification of all public APIs 3. Android - contains tests for the android APIs 4. Java - contains tests for the Java core library 5. VM - contains tests for the Dalvik virtual machine 6. RefApp - contains reference application tests (more coming in future CTS release)

Google Confidential

7. Performance - contains performance tests for your implementation (more coming in future CTS releases) These can be executed with the run cts command as mentioned earlier.

Google Confidential

5. Interpreting the Test Results

The test results are placed in the file: $CTS_ROOT/repository/results/<start time>.zip Inside the zip, the testResult.xml file contains the actual results -- open this file in any web browser (Firefox 3.x recommended) to view the test results.

The 'device information' section provides details about the device and the firmware (make, model, firmware build, platform) and the hardware on the device (screen resolution, keypad, screen type). The details of the executed test plan are present in the 'test summary' section which provides the CTS plan name and execution start and end times. It also presents an aggregate summary of the number of tests that passed, failed, time out or could not be executed. The next section also provides a summary of tests passed per package.

Google Confidential

This is followed by details of the the actual tests that were executed. The report lists the test package, test suite, test case and the executed tests. It shows the result of the test execution - pass, fail, timed out or not executed. In the event of a test failure details are provided to help diagnose the cause. Further, the stack trace of the failure is available in the XML file but is not included in the report to ensure brevity - viewing the XML file with a text editor should provide details of the test failure (search for the <Test> tag corresponding to the failed test and look within it for the <StackTrace> tag).

Google Confidential

6. Release Notes
6.1. General
Note the CTS test harness has changed signficantly in the Android 4.0 release. Some new features have been added included support for sharding a CTS test run onto multiple concurrent devices, as well as general faster performance. This CTS release contains approximately 17,000 tests that you can execute on the device. Please make sure all steps in section 4.2 "Setting up your device" have been followed before you kick off CTS. Not following these instructions may cause tests to timeout or fail.

6.2. Known Issues

Google Confidential

7. Appendix: CTS Console Command Reference

Host help help all Display a summary of the most commonly used commands. display the complete list of available commands Gracefully exit the CTS console. Console will close when all currently running tests are finished

exit Run run cts

Run the specified tests and displays progress information. One of --plan, -package, --class or --continue-session-id needs to be specified. The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS host will wait for a device to be connected before starting tests. If more than one device is connected, CTS host will choose a device automatically.

--plan <test_plan_name> Run the specified test plan -- package/-p <test_package_name> [-package/-p <test_package2>...] --class/-c <class_name> [--method/-m <test_method_name> --continue-session-id Runs all not executed tests from previous CTS session. The sessions testResult.xml will be updated with the new results. Runs the specified test packages. Runs the specified test class and/or method

Google Confidential

--shards <number_of_shards> Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel. --serial/-s <deviceID> Run CTS on the specific device List list packages List all available test packages in the repository. Lists all available test plans in the repository Lists 'run' commands currently being executed on devices. List all 'run' commands currently in the queue waiting to be assigned to devices List CTS results currently stored in repository List currently connected devices and their state. 'Available' devices are functioning, idle devices, available for running tests. list devices 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests. 'Allocated' devices are devices currently running tests. Add add derivedplan --plan <plan_name> --result/-r [pass | fail | timeout | notExecuted] [--session/-s <session_id>] Create a plan derived from given result session.

list plans list invocations list commands list results

Google Confidential

Google Confidential