How To Run Compatibility Test Suite (CTS) ?



How does the CTS work?

The CTS is an automated testing harness that includes two major software components:

The CTS test harness runs on your desktop machine and manages test execution.

Individual test cases are executed on attached mobile devices or on an emulator. The test cases are written in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.

Pre-Requirements :

  1. Linux System 
  2. Mobile or Emulator with User Build.
  3. Download latest CTS.
  4. Download CTS Media files.
  5. Download Android SDK for Linux and keep it ready. 

Setup the System for CTS:

1.Create 3 folders in your system root directory. 

  1. CTS
  2. CTS_Media
  3. ADT

2. Extract and Copy downloaded CTS files into the CTS folder which you have created now . It will have below folder structure 
    
     android-cts
               repository
         plans
                      results
         testcases
                      host_config.xml
      tools
3. Extract and Copy downloaded CTS Media files into the CTS_Media folder which you have created now .It will have below folder structure 
      
        1. Change the permission for copy_media.sh file
                $ export chmod 544 copy_media.sh

4.  Extract and Copy downloaded android-adt files into the ADT folder which you have created now .It will have below folder structure 

5. Open Terminal and Set ADT path by running below command form root directory.

    $ export PATH=$PATH:/home/root/ADT/android-sdk-linux/sdk/platform-toots
    Note : Run any adb command to check path is set properly or not. (exp:  adb devices )

Set up the Device/Emulator :

CTS can be executed only on consumer devices. This section is important as not following these instructions will lead to test timeouts/ failures:
1. Your device should be running a user build (Android 4.0 and later)from source.android.com
2. Please refer to this link on the Android developer site and set up your device accordingly.
3. Make sure that your device has been flashed with a user build (Android 4.0 and later) before you run CTS.
4. You need to ensure the Text To Speech files are installed on the device. You can check via Settings > Speech synthesis > Install voice data before running CTS tests. (Note that this assumes you have Android Market installed on the device, if not you will need to install the files manually via adb)
5. Make sure the device has a SD card plugged in and the card is empty. Warning: CTS may modify/erase data on the SD card plugged in to the device.
6. Do a factory data reset on the device (Settings > storage > Factory data reset). Warning: This will erase all user data from the device.
7. Make sure no lock pattern is set on the device (Settings > Security > Screen Lock should be 'None').
8. Make sure the "USB Debugging" development option is checked (Settings > Developer options > USB debugging). Google Confidential
9. Make sure Settings > Developer options > Stay Awake is checked
10. Make sure Settings > Developer options > Allow mock locations is checked
11. Make sure device is connected to a functioning Wi-Fi network (Settings > Wi-Fi)
12. Make sure the device is at the home screen at the start of CTS (Press the home button).
13. While a device is running tests, it must not be used for any other tasks.
14. Do not press any keys on the device while CTS is running. Pressing keys or touching the screen of a test device will interfere with the running tests and may lead to test failures.
15. Set up accessibility tests:
   1. adb install CTS/android-cts/repository/testcases/ CtsDelegatingAccessibilityService.apk
   2.On the device, enable Settings > Accessibility > Delegating Accessibility Service
16. Set up device administration tests: 
  1. adb install CTS/android-cts/repository/testcases/CtsDeviceAdmin.apk
  2. On the device, enable Settings > Security > Device Administrators > android.deviceadmin.cts.CtsDeviceAdmin* settings

17. Copy the CTS media files to your device using ADB with the instructions below:
   The CTS media stress tests require video clips to be on external storage (/sdcard). Most of the clips are from Big Buck Bunny which is copyrighted by the Blender Foundation under the Creative Commons Attribution 3.0license. The required space depends on the maximum video playback resolution supported by the device. By default, 176x144 and 480x360 SHOULD be supported. Note that the video playback capabilities of the device under test will be checked via the android.media.CamcorderProfile APIs.
Here are the storage requirements by maximum video playback resolution:
  • 480x360: 91.4MB
  • 720x480: 151.9MB
  • 1280x720: 401.6MB
  • 920x1080: 1008.2MB

Instructions to copy the media files to a device:
  • To copy clips for just the default resolutions, run ./copy_media.sh. Google Confidential 
  • To copy clips up to a resolution of 720x480, run ./copy_media.sh 720x480. 
  • If you are not sure about the maximum resolution, try 1920x1080 so that all files are copied. ○ If there are multiple devices under adb, add -s serial option to the end. 
Syntax  :  $.<space>./CTS_Media/copy_media.sh <resolution>
Example :  $. ./CTS_Media/copy_media.sh 1920x1080


For example, to copy up to 720x480 to device with serial 1234567, run copy_media.sh 720x480 -s 1234567.
Now , Start Using CTS commands :

1. Run the below command to start cts . 
   Syntax  $./CTS/tools/cts-treadfed run cts --plan <plan-name>
   Command $./CTS/tools/cts-treadfed run cts --plan CTS
    
     Note : It will take around 10 hours to complete test and it will create report in CTS/android-cts/repository/results folder.  To list the reports follow the below step.

2. Run the below command to list results . 
   Command $./CTS/tools/cts-treadfed list results
      Results will be shown as below :


3. Run the below command to continue the session if there is a "NotExecuted" cases  . 
      Syntax   $./CTS/tools/cts-treadfed add derivedplan --p <new-plan-name> --s <session-id> --r notexecuted
   Command $./CTS/tools/cts-treadfed add derivedplan --p CTS-FAIL --s 0 --r notexecuted

4. Run the below command to create new plan with Failed cases . 
      Syntax   $./CTS/tools/cts-treadfed add derivedplan --p <new-plan-name> --s <session-id> --r fail 
   Command $./CTS/tools/cts-treadfed add derivedplan --p CTS-FAIL --s 0 --r fail

5. Run the below command to create new plan with Passed cases . 
       Syntax   $./CTS/tools/cts-treadfed add derivedplan --p <new-plan-name> --s <session-id> --r pass
   Command $./CTS/tools/cts-treadfed add derivedplan --p CTS-FAIL --s 0 --r pass

6. Run the below command to run individual methods in class . 
       Syntax   $./CTS/tools/cts-treadfed run cts --class <class-name> --method <method-name>
      Command $./CTS/tools/cts-treadfed run cts --class android.permission.cts.FileSystemPermissionTest --method testOtherRandomDirectoriesNotWritable


7. Run the below command to run all methods in class . 
       Syntax   $./CTS/tools/cts-treadfed run cts -c <class-name>
       Command $./CTS/tools/cts-treadfed run cts -c android.permission.cts.FileSystemPermissionTest

Comments

  1. Thanks in favor of sharing such a fastidious idea, piece of writing is nice, thats why i have read it…

    couples compatibility test for two

    ReplyDelete
    Replies
    1. hi brother have you got answer for this ?,
      if got please help me out by sharing

      Delete
  2. I have read your blog its very attractive and impressive. I like it your blog.

    Java Training in Chennai Java Training in Chennai | Core Java Training in Chennai

    Online Java Training Online Java Training | Java J2EE Online Training | JavaEE Training Institute in Chennai m

    ReplyDelete
  3. hi thank you soo soo much
    your explanation with ex. is very good
    from 1 day i was searching for continue session now i got
    really thanks for your kind sharing

    ReplyDelete
  4. Hi
    How to run multiple modules at a time or under one plan? Please explain with example.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...
    ios App Development Company
    Mobile App Development Company
    Best Mobile App Development Company

    ReplyDelete

Post a Comment

Popular posts from this blog

NFC : Reading a MiFare Classic 1K from Android Devices

What is the directory structure of android AOSP root tree?