Abstract
In applied research in communication and journalism, as well as in other related sciences, it is common to use tests to assess unobservable constructs. The scores of these tests frequently need to be given meaning and interpreted, and their psychometric properties need to be reported as part of the study process or because the peer review procedure requires it. This chapter reviews the validity evidence required to give meaning to the scores of a study. It also provides practical examples from the literature, material for understanding the techniques to be applied, and an overview of best practices when using tests.
TopBackground
Despite the naïve assumption that the inclusion of a test is a routine action, the use of tests for research is much more complex than one might imagine. Researchers often believe that as long as a test has been validated and published in the past (either in a test editorial or in previous research articles) there is no reason not to trust it. However, this rationale exposes researchers and practitioners to lack of psychometric guarantees in their selected tests. This potential thoughtlessness towards measurement is broader that just test selection. The recently named “credibility revolution” (Vazire, 2018; also called “replication crisis,” Nelson et al., 2018) signals that the scientific community is beginning to question the credibility of their findings, in which perhaps one of the biggest problems is related to the measurement of concepts (Flake & Fried, 2020; Scheel et al., 2021).
Or, put another way, are we sure we measure what we want to measure? Within this revolution, questionable or sub-optimal practices are used to measure unobservable concepts, which encompass from uninformed or not exhaustive review of psychometric literature to thoughtless application of statistical techniques (e.g., Viladrich et al., 2017; Navarro et al., 2018, chapter 8; an applied example in Lecuona et al., 2020). Thus, those who are doing the measuring need to ask what they can do to improve the process. Every decision that is made both when developing a test and when adapting it to certain research can jeopardize the interpretation of the scores. This is not a new issue. Forty years ago, McCroskey and Young (1979) were already warning about the misuse of certain techniques in communication studies (especially in the context of SEM [Structural Equation Modeling]).
Key Terms in this Chapter
Convergent Validity: Evidence that is provided when two tests have a similar or close meaning. It is good evidence if both concepts that are theoretically related show a strong relationship; it is bad evidence if they are not related.
Factor Analysis: A type of structural equation modeling that is used to study the relationship between observed indictors (items) and latent variables (factors). The most well-known models are exploratory factor analysis and confirmatory factor analysis.
Structural Equation Modeling: Set of techniques used to assess the relationships between observed (items) and unobserved (factors) variables. This methodology is popular in behavioural and health sciences.
Criterion Validity: Evidence that is provided when there is an external variable (or a group of external variables) that can be predicted with the test.
Psychometric Networks: Visual statistical technique used to represent relationships between observed variables explaining the relationship between psychological constructs.
Differential Item Functioning: This is said to occur when, at the same level of ability in a construct, there is a different probability of giving a correct answer that can be explained due to aspects that are not related to this construct but are related to membership of a group (e.g., gender, socioeconomic status, or race).
Discriminant Validity: Evidence that is provided when two tests aim to measure independent constructs. It is good evidence if there is a null to weak relationship between both constructs; it is bad evidence if they are found to be related (when this is theoretically unexpected).