Stealth Assessments' Technical Architecture

Introduction

Learning and engagement theories—such as the zone of proximal development (Vygotsky, 1978) and flow (Csikszentmihalyi, 1990)—suggest that challenges in a learning environment should match learners’ ability. With advances in technology, as well as in the learning and assessment sciences, educators can develop learning environments that can accurately assess and support learners’ knowledge, skills, and other attributes (Shute et al., 2016; Shute & Rahimi, 2017). Such learning environments rely on real-time competency estimates to adapt challenges to learners’ ability levels or to provide appropriate supports to maximize learning.

Stealth assessment (Shute, 2011) uses games or other technology-rich environments as a vehicle to assess learners’ emerging competencies (e.g., creativity, problem-solving, physics understanding). Stealth assessment is based on an assessment design framework called evidence-centered design (ECD; Almond et al., 2002). ECD allows stealth assessment designers to define the competency (unobservable) they are interested to assess (i.e., the competency model), identify good in-game indicators (observables), which can be statistically linked to the competency model (i.e., the evidence model), and define and create tasks that can elicit the evidence needed for the evidence model (i.e., the task model). When these three core models are established and implemented in a system, observations made in the context of stealth assessment tasks provide evidence of competency levels, allowing the system to update competency estimates in real-time.

The ongoing performance data are collected in log files as a learner interacts with the game. The stealth assessment then automatically scores and accumulates the collected data using statistical methods (e.g., Bayes nets), and makes real-time inferences about the learner’s current level of targeted competencies, adapting the game difficulty accordingly (see Rahimi et al., in press; Smith et al., in press for more details). To make stealth assessment possible, various technical components need to seamlessly work together. The purpose of this chapter is to describe the various components (i.e., the architecture) of an adaptive environment in the context of a game called Physics Playground (PP; Shute et al., 2019). PP’s architecture can be used to create other educational games equipped with stealth assessment.

PP is a 2D game with simple game mechanics (e.g., drawing lines and creating physics simple machines; see Figure 1). The goal in this game is to direct a green ball to a red balloon. There are two level types in PP: sketching and manipulation. To solve sketching levels, learners draw simple machines (i.e., ramps, levers, pendulums, and springboards) to guide the ball to the balloon (Figure 1a). To solve manipulation levels, learners interact with various sliders to change physics parameters (i.e., gravity, air resistance, mass, and bounciness of the ball), and also manipulate external forces exerted from puffers or blowers (Figure 1b).

Figure 1.

Through an iterative process, we designed hundreds of game levels (explained later) and developed numerous learning supports in PP to assess and enhance learners’ physics understanding related to specific concepts (Figure 2). Learners earn a gold coin for an elegant solution using a small number of objects or attempts, a silver coin for a regular solution with more than a predefined number of objects or attempts, or nothing for a failed attempt.

Figure 2.