Would it be a good idea to have an interview in a format of a paired exploratory testing session with a candidate on a manual/automated testing position? A thing under test might be an internal web application or a publicly opened web resource.
What things to look for in a candidate and how to quantify the results of such an interview? (I don't think that number of newly-found bugs is a good and relevant measurement)
Yes, it would. I like teams that execute exploratory testing in pairs, I think it is easier to write charters, create mind-maps and decide on heuristics together than alone. It will keep you sharper, you will go faster in executing and analysing the results.
I would like to find answers on the following questions during an interview:
Seeing how someone would approach a pair-testing session is worth a lot. It is about communication and test design. Also, you will notice if someone is a quick learner. Can they lead the session and get some results while working together with someone else. How do they respond to critical feedback during the session.
Exploratory testing sessions are time-boxed, making it a great candidate for a second interview test. You can perform it with a time-box of let's say 45 minutes and work with a tester colleague. Afterwards, review the results together with the hiring manager. Giving the candidates room to explain what they would do differently next time for example. Resulting in an interview of around 1.5 hours, which I think would be acceptable.
Alternatively, I would do pair-programming of test automation on a simple application. Just like I would always do a pair-programming interview for developers.
Great idea. Some things that might help:
Additionally, if this person is going to do test automation, I've listed some questions you can ask here.
I have done something like this for a manual tester position. I whipped up a quick MVC site with 3 pages and put "issues" into each page (something like 11 issues total). I made it clear that I wasn't looking for the interviewee to find all the issues. I just wanted to watch how they go about it.
I sat with each candidate for 30 minutes and tried to be as friendly and helpful as possible. This is a high stress situation, which is also something I was looking for the candidate to handle.
In the end I only did this once as it was very time consuming and I started to only fill more Sr level positions on my team.
Quite interesting topic and although this question has been raised two years ago it is still a current topic! Hence I would like to share our experience / thoughts.
Preparation from our side as business department:
What did we learned from paired exploratory testing & what things to look for in a candidate?
I wouldn't refuse number of found issues as a relevant measurement. After all you're looking for an engineer that would be finding defects in your applications.
As to me, any interview should spot a light onto two kind of person's skills: hard and soft ones. I tend to consider such kind of interview spotting both types: on one hand a person shows you how they would test the particular component (and this is your responsibility to pick the representative application for an interview), on other hand the person shows how they would comunicate their thoughts to teammates.
Since you cannot measure (using any objective scale) a person's soft skill, you can only rely on:
However if this is the only method you're going to apply in your interview you should take care of the "quality" of the application for the interview since it might not cover all the representative cases that one would meet in the real project life.
This is a very interesting topic and I have conducted a few interviews similar to this and here is my feedback based on the interviews:
However, these are not only the points, but you can also evaluate the candidates on other criteria based on the position requirements.
I think this is a much better idea than asking questions such as "Where do you see yourself in 5 years?" etc. :)
Let me share a few points I find important:
Regarding the application under test. I'd personally choose one I know and one where at least some bugs could be found. That's because I'd like to see the candidate reporting issues or at least talking about (possible) issues as well.
The session should have a particular time slot, for your situation, I'd go for something in the range of 30 - 60 minutes.
Before even starting and/or presenting this to the candidate, I'd let them share their ideas about such exploratory sessions and how they think this could be done. You can find out from their answers if they know how to orginise such exploratory sessions.
During the session, let them talk. It's important because you can see how they communicate, how they go about exploration, how they discuss problems, how they ask. A tester should be able to communicate well, and this is a way to find out.
I'm a bit sceptical about quantifying the results. Why do you need to do this? Is it because HR asks you to? Just like exploratory testing cannot be fully quantified, and you don't want it to be, I wouldn't necessarily try to make the interview quntifiable. If you really need to quantify something, it shouldn't be the number of bugs. I'd rather create a checklist of imporatant things/skills I want the tester to know (and I can find out about during the exploratory session and interview) and count the number of checks I make with each candidate. Or you can create your own scale with well-defined levels of a skill. You probably might ask HR about how to go about this, I assume they have some experience in this area.
All in all, I think that seeing a candidate at work brings valuable information about how they fit in your team/company. Such a session is a short peek into exactly this very thing.
