The evaluation of ubiquitous computing (Ubicomp) applications presents a number of challenges ranging from the optimal recreation of the contextual conditions where technologies will be implemented, to the definition of tasks which often go well beyond the model of human-computer interaction that people are used to interact with. Many contexts, such as hospitals and healthcare which are frequently explored for the deployment of Ubicomp raise additional challenges as a result of the nature of the work where human life can be in risk, privacy of personal records is paramount, and labor is highly distributed across space and time. For the last six years the authors have been creating and pilot-testing numerous Ubicomp applications in support of hospital work and healthcare (Ubihealth). In this chapter, they discuss the lessons learned from evaluating these applications and organize them as a frame of techniques that assists researchers in selecting the proper method and type of evaluation to be conducted. Based on that frame the authors discuss a set of principles that designers must consider during evaluation. These principles include maintaining consistency of the activities and techniques used during the evaluation, give proper credit of individual benefits, promote replication of the environment, balance constraints and consider the level of pervasiveness and complexity.
TopIntroduction
The ubiquitous computing vision aims at surrounding users with a comfortable and convenient information environment that merges physical and computational infrastructures into an integrated habitat (Weiser, 1991). As Weiser stated “the most powerful things are those that are effectively invisible in use […] They weave themselves into the fabric of our everyday life until they are indistinguishable from it“ (Weiser, 1998). Indeed, Ubicomp technologies may be embedded in the environment, embedded in objects, worn, or carried by the user throughout everyday life. Interactions may be implicit or even sensed, unlike the explicit and intentional interactions typical of user interfaces in desktop environments. Ubicomp technologies may be used in unpredictable situations, changing contexts, and highly mobile applications. These issues and the invisibility posed by Weiser’s vision alter the relationship between humans and technology posing multiple challenges for the evaluation of ubicomp.
While there are currently available an ample set of techniques and examples of ubicomp evaluation following traditional HCI approaches (Consolvo et al., 2007), using traditional techniques for evaluating ubicomp systems is quite challenging as we need to consider other issues. Among these issues include the ambiguity, scalability, and context of use of an ubiquitous application (Carter & Mankof, 2004). For example, a context-aware system can assume that every object of interest will be tagged with an RFID to automatically adapt the information shown to a user. Researchers have only recently begun to address the development of evaluation techniques that meet ubicomp’s demands (Carter & Mankof, 2004; Consolvo et al., 2007; Mankoff et al., 2003). One reason for this is the relatively slow development and gradual evolution of methods and techniques that allow researchers to evaluate ubiquitous applications at different stages of their implementation. Among those efforts we can found the Wizard of Oz as one of the most popular techniques adopted (Consolvo et al., 2007). This distance from the real work practice and context means that much has to be done to understand how to apply qualitative or quantitative evaluation methods for Ubicomp.
The challenges to evaluate Ubicomp are emphasized in a healthcare domain. Hospitals represent a fruitful area to study the evaluation of Ubicomp and many researchers have focused on providing solutions for this particular context. Indeed, some elements of ubicomp are gradually being introduced in hospitals. These range from wireless networks, PDAs (Chin, 2005), RFID tags for patient tracking (O'Connor, 2006), voice-activated communication devices (Stanford, 2003), and sensors for patient monitoring (Pentland, 2004). However, given the critical nature of hospital work, the creation of such an Ubihealth environment needs to be gradual. This will allow confidence on the technology to build and lessons-learned to be incorporated into such environment. These issues in Ubihealth raise new challenges that go beyond those posed by ubicomp. Those challenges involve the level of integration of an Ubihealth application with existing systems deployed in the hospital, the context of use of such system that must be replicated when evaluating the system, as well as, the level of risk involved in case of system failure. As others researcher have pointed before, we regret the lack of techniques that allow us to cope with the evaluation issues faced when evaluating Ubihealth systems.
In this chapter we discuss our lessons learned from evaluating three Ubihealth applications. We organize our lessons a set of design principles that assist researchers in selecting the proper method and type of evaluation to be conducted. The rest of the chapter is organized as follows: In section II we describe the related work in the area highlighting the challenges for Ubicomp evaluation and why those are more difficult in the healthcare domain. Section III, describes six case studies of the evaluation of three systems. In section IV, we discuss how our lessons learned were organized as a set of principles that allow designers to manage those challenges in the evaluation of Ubihealth applications. Finally, in section V we present our conclusions and directions for future work.