How To Use Accessibility Services For "taking Action For Users"?

Background Back a few years ago, I asked how TeamViewer allows the user to control the device without normal interaction with the device. I was told it's a special 'backdoor' that

Solution 1:

By implementing AccessibilityService (https://developer.android.com/training/accessibility/service.html) you get access to that features.

You can either inspect or perform action on the element lastly interacted by user or inspect whole application which currently active.

Intercept user events by implementing onAccessibilityEvent(AccessibilityEvent event), here you can retrieve virtual view (representing original view) with event.getSource() and then inspect it with getClassName() or getText() or anything you find in the documentation.

Inspect whole application by calling getRootInActiveWindow() and iterate throught tree of virtaul views with getRootInActiveWindow().getChild(index).

Both getRootInActiveWindow() and event.getSource() return AccessibilityNodeInfo, on which you can invoke performAction(action) and do something like Click, Set Text, etc..

Example: Play Store

Search for 'facebook' app and open it's page on play store, once you opened the play store app.

@OverridepublicvoidonAccessibilityEvent(final AccessibilityEvent event) {

        AccessibilityNodeInforootInActiveWindow= getRootInActiveWindow();
        //Inspect app elements if readyif (rootInActiveWindow != null) {
            //Search bar is covered with textview which need to be clicked
            List<AccessibilityNodeInfo> searchBarIdle = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_idle_text");
            if (searchBarIdle.size() > 0) {
                AccessibilityNodeInfosearchBar= searchBarIdle.get(0);
                searchBar.performAction(AccessibilityNodeInfo.ACTION_CLICK);
            }
            //Check is search bar is visible
            List<AccessibilityNodeInfo> searchBars = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_text_input");
            if (searchBars.size() > 0) {
                AccessibilityNodeInfosearchBar= searchBars.get(0);
                //Check is searchbar have the required text, if not set the textif (searchBar.getText() == null || !searchBar.getText().toString().equalsIgnoreCase("facebook")) {
                    Bundleargs=newBundle();
                    args.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, "facebook");
                    searchBar.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args);
                } else {
                    //There is no way to press Enter to perform search, so find corresponding suggestion and click
                    List<AccessibilityNodeInfo> searchSuggestions = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/suggest_text");
                    for (AccessibilityNodeInfo suggestion : searchSuggestions) {
                        if(suggestion.getText().toString().equals("Facebook")) {
                            //We found textview, but its not clickable, so we should perform the click on the parentAccessibilityNodeInfoclickableParent= suggestion.getParent();
                            clickableParent.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        }
                    }
                }
            }


        }
   }

---

EDIT: full code below:

MyAccessibilityService

publicclassMyAccessibilityServiceextendsAccessibilityService {

    @OverridepublicvoidonCreate() {
        super.onCreate();
        Log.d("MyAccessibilityService", "onCreate");
    }

    @OverridepublicvoidonAccessibilityEvent(final AccessibilityEvent event) {
        Log.d("MyAccessibilityService", "onAccessibilityEvent");
        AccessibilityNodeInforootInActiveWindow= getRootInActiveWindow();
        //Inspect app elements if readyif (rootInActiveWindow != null) {
            //Search bar is covered with textview which need to be clicked
            List<AccessibilityNodeInfo> searchBarIdle = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_idle_text");
            if (searchBarIdle.size() > 0) {
                AccessibilityNodeInfosearchBar= searchBarIdle.get(0);
                searchBar.performAction(AccessibilityNodeInfo.ACTION_CLICK);
            }
            //Check is search bar is visible
            List<AccessibilityNodeInfo> searchBars = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_text_input");
            if (searchBars.size() > 0) {
                AccessibilityNodeInfosearchBar= searchBars.get(0);
                //Check is searchbar have the required text, if not set the textif (searchBar.getText() == null || !searchBar.getText().toString().equalsIgnoreCase("facebook")) {
                    Bundleargs=newBundle();
                    args.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, "facebook");
                    searchBar.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args);
                } else {
                    //There is no way to press Enter to perform search, so find corresponding suggestion and click
                    List<AccessibilityNodeInfo> searchSuggestions = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/suggest_text");
                    for (AccessibilityNodeInfo suggestion : searchSuggestions) {
                        if (suggestion.getText().toString().equals("Facebook")) {
                            //We found textview, but its not clickable, so we should perform the click on the parentAccessibilityNodeInfoclickableParent= suggestion.getParent();
                            clickableParent.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        }
                    }
                }
            }
        }
    }

    @OverridepublicvoidonInterrupt() {
    }
}

AndroidManifest.xml

<manifestxmlns:android="http://schemas.android.com/apk/res/android"package="com.example.findfacebookapp"><applicationandroid:allowBackup="true"android:icon="@mipmap/ic_launcher"android:label="@string/app_name"android:supportsRtl="true"android:theme="@style/AppTheme"><activityandroid:name=".MainActivity"><intent-filter><actionandroid:name="android.intent.action.MAIN"/><categoryandroid:name="android.intent.category.LAUNCHER"/></intent-filter></activity><serviceandroid:name=".MyAccessibilityService"android:label="@string/accessibility_service_label"android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"><intent-filter><actionandroid:name="android.accessibilityservice.AccessibilityService"/></intent-filter><meta-dataandroid:name="android.accessibilityservice"android:resource="@xml/accessibility_service_config"/></service></application></manifest>

res/xml/accessibility_service_config.xml

<?xml version="1.0" encoding="utf-8"?><accessibility-servicexmlns:android="http://schemas.android.com/apk/res/android"android:accessibilityEventTypes="typeAllMask"android:accessibilityFeedbackType="feedbackAllMask"android:accessibilityFlags="flagDefault"android:canRequestEnhancedWebAccessibility="true"android:canRetrieveWindowContent="true"android:description="@string/app_name"android:notificationTimeout="100"/>

MainActivity

publicclassMainActivityextendsAppCompatActivity {

    publicvoidonEnableAccClick(View view) {
        startActivityForResult(newIntent(Settings.ACTION_ACCESSIBILITY_SETTINGS), 1);
    }

}