6

There should be some Key Performance Indicators designed for measuring AI performance. For example, the number of entities examples you have to feed it in order to obtain single task on a testing entity with repeatable 97% accuracy.

Is there any of such measure constructed?

Motivation: you cannot learn AI to be good at playing a game released 1 month ago because there is not enough data on how to play. It doesn't matter how clever you are, present brute force paradigm in AI just doesn't fit in such circumstances.

DukeZhou
  • 6,227
  • 5
  • 25
  • 53
kakaz
  • 271
  • 1
  • 6
  • 1
    Welcome to AI! I feel this needs a slight qualification in that game complexity is a determining factor. If complexity is low, brute force is more than adequate. If the game is non-trivial, solution and AI performance becomes an issue. Great question though. – DukeZhou Sep 11 '17 at 16:40

1 Answers1

3

The KPI's would probably be different for each methodology. For example an NB classification algorithm can easily reach %97 and above (and it would be very easy to measure this) but it would be difficult to measure say a game playing algorithm. Would you suggest counting the wins/losses or another way to measure them?

solarflare
  • 324
  • 2
  • 10
  • Take a 4,5 years old child for example. It takes 2 or 3 times to point on a cat and a child is able to recognise it. Take Google for example. 10^7 cat pictures is probably nearly enough to learn Google how cat look like. I would like to measure this gap. – kakaz Sep 11 '17 at 15:54
  • 1
    It would entirely depend on what algorithm you chose to use. I'm a huge fan of NB (personal choice) as it is easy to construct and works surprisingly well compared to more sophisticated methods. Try using Weka and play around with the differences in algorithms, it will also provide the statistics you are looking for. You can download it for free from the Waikato Uni website. – solarflare Sep 13 '17 at 05:51