Designing Games for Stealth Assessment: Dos and Don'ts of the Trade Based on Practical Experience

Introduction

For me, the idea of using games to inquire about learning goes back to when I was a child and overheard an aunt telling my mum how she observed her children play video games. She said, “They make mistakes, think about them, try a different solution, and eventually learn how to overcome the obstacle and interiorise the strategy to advance”. At that time, this made me think that what I was doing when playing “Atari” was not a mere pastime but could even be educational.

More recently, I was finishing my studies in psychology. I had one year to carry out my thesis, and I knew I wanted to do it on video games. I studied the task analysis method proposed by Newell and Simon (1972) for a while. My idea was straightforward; I wanted to use task analysis with games to examine how people solve problems. The purpose behind this was to use games to help overcome problems intrinsic to traditional psychological and educational evaluations. These types of problems include evaluation anxiety effects (roughly defined as the fear of the negative consequences product of an evaluation. Asghari et al., 2012; Rhamn, 2016) as well as the Pygmalion effect, which is when participants improve their performance in an evaluation according to the evaluator’s expectations (Kalyanasundaram, 2017; Reynolds, 2007).

Identifying the use of games for assessment purposes (as is the case with stealth assessment) was not a straightforward task. There was a dearth of literature about the development and data tracking process related to stealth assessment while writing my psychology thesis. Inspirations and examples were Griffiths' (2002) paper, where he proposes games as educational research tools. Navarro-Roldán (2008) designed a game to understand how children reason about cogs' operation, and Shute et al. (2008) report on embedded assessment. Finally, Puche-Navarro and Combarisa (2009) developed a gamified task for mathematically understanding and modelling children’s development of humour. In addition, I looked extensively for people who had done something similar and asked different experts, finding the most similar works to be those of Navarro-Roldán and Puche-Navarro, with whom I made my psychology internship. So, similar approaches to the one I wanted to implement were scarce.

Given this, I decided to go it alone. The first thing that I did was design a two-layered game. The first layer was the game itself; the second layer was a database capturing (in real-time) the data generated by players’ actions in the game. By having a two-layered system, players could have fun solving game-based tasks, while I was able to obtain data allowing me to observe, analyse, and model the process by which players solved problems.

Around the same time, in 2010, a commercial game was released that claimed to do the same thing that I was doing in my own game. That game, Silent Hill: Shattered Memories (Silent Hill: Shattered Memories. Gameplay, 2016), claimed to collect data on its players' behaviours and, from this process, generate a psychological player profile. This idea really excited me! Was it truly possible to use players' interactions within a game to create psychological profiles? Thus, Silent Hill opened new ideas and reasons for using stealth assessment and sparked ideas about how it could be done, and I was ready to take the challenge to develop them!

In 2013, I was deeply involved in my doctoral studies at the Ludwig-Maximilians-Universität München. At this time, I was attempting to analyse how players use scientific reasoning when they play Multiplayer Online Battle Arenas (Díaz, 2017) using the same techniques I used during my psychology degree and my master's. Around that same time, I learned about the ground-breaking work of Valerie Shute (Shute & Spector, 2010). Shute and her team were doing what I had been trying to do during the last years of my career. But importantly, her team systematically explained the steps and named them within the overall framework of stealth assessment.