9

Given a screenshot of an application, is there any way of finding GUI elements of that application using only opencv? (something like in the image) Desired Outlook image after element detection

I tried to play with blending the image and other effects from the coherence.py sample in the opencv installation folder. Did not succeed to get a viable result.

Has anyone did this? Do you have any idea what I might try?

Thanks

(I'm new to opencv and image processing so please don't avoid obvious explanations if possible) :)

Radu Enea
  • 185
  • 2
  • 2
  • 5
  • 2
    This is not in OpenCV, but have a look at Sikuli.org which has exactly the purpose you're looking for – Ivo Flipse Mar 08 '12 at 13:41
  • i know about Sikuli. What i needed was a way to detect the areas of interest without actually taking a screen-shot of them in advance. The answer was the MSER approach with specific initialization. Anyway, thanks for the Sikuli tip. – Radu Enea Mar 09 '12 at 14:17
  • If I may ask what were you programming? I got here searching for the same question. My intention is to make elements keyboard-clickable when for applications that don't have keyboard shortcuts defined. Did you worked on something similar? – urza.cc Mar 15 '17 at 07:20
  • No, it was for a small project of UI test automation. It was supposed to "watch" the user interact with the UI and then generate small scripts to replay the actions. I abandoned it at some point due to Sikuli's lack of reliability on long term. I could get screenshots of the elements in UI, but Sikuli did not manage to find all of them in 100% of cases (probably they were too small or too similar to other elements) – Radu Enea Apr 03 '17 at 10:30

1 Answers1

10

OpenCV's MSER extractor (documented here) might be helpful — the bounding box of local MSER groups would pretty closely match the green rectangles in your mockup.

smokris
  • 557
  • 1
  • 6
  • 7