In what way to check that a multi-platform software is working as expected?

Question

Let's say we have a chat software that can be used via:

1. a mobile application
2. standalone app
3. web application

A User using one of this platform can use the software to communicate with another User in another platform.

When we talk about End to end scenario, shall we create automatic UI test so that we make sure that every User with every device can communicate with every other User with other devices? Shall we also cover the case that the system is working well using a big number of Users?

I believe that the E2E test, in this case should cover only the bare minimum communication among the three devices, no more than that, given the reliability and the speed of the test, and I would delegate the health check of the connection between the services to some monitoring tool, so that in case of a failure we are able to detect immediately if the system can work or not.

Answer 1

"shall" is different from "should". Like everything in engineering, it is always about feasibility: cost vs benefits.

Creating automated test which run on 3 different platforms and synchronize runs between themselves (using of course separate pathway than you are testing) is possible, but likely quite complicated.

OTOH, running the same test by 3 manual testers sitting in the same room is trivial.

So you need to balance the cost of developing the automated solution vs the cost of doing it manually, and how much more bugs will be prevented by fully automated solution.

Certainly there could be situation where only the fully automated solution is acceptable, but for most situations it would be prohibitively expensive.

Answer 2

You should cover the specific devices you wish to test 'at one end' of the communication and for the other end you supply a fake device or data. You want to test how each devices deals with info it send and receives but you should not teach every 'device combination' as that doesn't add more value and when you have 8 devices / platforms that means 8144 device combinations...

A big number of users requires a different sort of testing. That is volume testing and is commonly done with tools such as jmeter or through in-house built solutions.