Android Clickable On screen text

Question

I was curious, I seen this app the other day that allowed it to open other apps and set certain functions up for you automatically. I have came to realize that it must be using an on screen click function of some sort, but I can't seem to find any documentation for something like this. For example if we know the on screen text from the other app is "Ready", is there a way to read that text and maybe do something like:

protected void processText(String text)
{
  if (text.contains("Ready"))
      // click the ready text
}

Answer 1

I have done this using AccessibilityService. It will only work fine on API level >= 16 though.

You need to extend AccessibilityService. For instance this class will get text of USSD responses and dismiss the dialog.

// ....
public class UssdAccessibilityService extends AccessibilityService {
    public UssdAccessibilityService() {
    }

    @TargetApi(16)
    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
        if (!"com.android.phone".equalsIgnoreCase((String)event.getPackageName())){
            // In this example we are only interested in events comming
            // from "com.android.phone" package
            event.recycle();
            return;
        }

        String className = (String)event.getClassName();
        if (className == null || (!className.contains("AlertDialog") && !className.contains("AlertDialog"))){
            // Class is not an USSD dialog
            event.recycle();
            return;
        }

        AccessibilityNodeInfo source = event.getSource();
        if (source == null) {
            // getSource() is annotated @Nullable, so we do this to be
            // safe just in case
            event.recycle();
            return;
        }

        AccessibilityNodeInfo acceptButton = null;
        String ussdText = null;

        int childCount = source.getChildCount();
        for (int i = 0; i < childCount; i++){
            AccessibilityNodeInfo current = source.getChild(i);
            if (current == null)
                continue;

            String currentText = (String)current.getText();
            if (current.isClickable()){
                // In the case of USSD dialogs, there is only one clickable.
                // May be necessary to do more robust search in other scenarios
                acceptButton = current;
                continue;
            }

            ussdText = currentText;

            current.recycle();
        }

        if (ussdText!= null) {
            if (acceptButton != null)
                acceptButton.performAction(AccessibilityNodeInfo.ACTION_CLICK);

        }

        source.recycle();
        event.recycle();
    }

    // ....
}

You must declare the accessibility service in the manifest under <application>

<service
        android:name=".UssdAccessibilityService"
        android:enabled="true"
        android:label="Read USSD codes and dismiss"
        android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
        <intent-filter>
            <action android:name="android.accessibilityservice.AccessibilityService" />
        </intent-filter>

        <meta-data
            android:name="android.accessibilityservice"
            android:resource="@xml/accessibility_service_config" />
</service>

Under res/xml create accessibility_service_config.xml

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:packageNames="com.android.phone,com.ats.android.activationcodebot"
    android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
    android:accessibilityFlags="flagDefault"
    android:accessibilityFeedbackType="feedbackSpoken"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
    />

Of course you have to adapt this code to your own needs.

Finally you will have to enable the accessibility service manually on Settings > Accessibility in Android (or ask the user to do it).

Read more ... Developing an Accessibility Service

Answer 2

To be able to do that from another app you need to capture the screen and determine the text position using text recognition service.

when another application is active, then your application should be active to be able capture the screen. So you can only use android service which always works background.

However to capture ScreenShot for your activity you have to need a View of your activity, and which one is not present in your service so you have to make a TimerTask which will call your activity at every hour and your activity responding it to with current appear view and you can capture the ScreenShot from that.

Or If you want to take a ScreenShot of your current device screen (any application) then you have to root permission, and read framebuffer for that which will give raw data of current screen then convert it to bitmap or any picture file you can do it in your service.

Briefly, It may possible for rooted phones. Otherwise, You can't take screenshot of other apps if you are non rooted. Allowed only to take screenshot of your app.

To be able analyze the screen whether there is any "Ready" text or not. Your application should be lived/actived to be able to capture of the screen.