Social impact assessment is a fundamental process to verify the achievement of the objectives of interventions and, consequently, to validate investments in the social area. Generally, this process is based on the analysis of the average effects of the intervention, which does not allow a detailed understanding of the individualization of these effects. Causal machine learning methods mark an evolution in causal inference, as they allow for a more heterogeneous assessment of the effects of interventions. Applying these methods to evaluate the impact of social projects and programs offers the advantage of improving the selection of target audiences and optimizing and personalizing future interventions. In this chapter, in a non-technical way, the authors explore classical causal inference methods to estimate average effects and new causal machine learning methods to evaluate heterogeneous effects. They address adapting the Uplift Modeling method to assess social interventions. They also address the advantages, limitations, and research needs for using these new techniques in social intervention.
Top1. Introduction
Recently, due to the emergence of ChatGPT and similar technologies, there has been much talk about Artificial Intelligence (AI) and Machine Learning (ML). As we explore the potential of AI/ML, we are seeing significant advances in natural language processing, neural networks, generative networks, computer vision, and others. However, despite these advances, it is only recently that the scientific community has begun to look more deeply into the relationship between Causality and AI/ML. This work focuses precisely on this interconnection. The text is divided into two distinct parts: a brief introduction to the classical approach to causal inference, with an emphasis on econometrics, followed by a second, more specific part on how to apply AI/ML to causal inference. We will explore the advantages of these new approaches for a more complete understanding of causality, focusing on the practical application of causal inference in social impact assessment.
According to a report by the Organization for Economic Cooperation and Development (OECD) published in 2021, the need for better management of funding for social programs and projects has led to an increasing number of funders requesting impact evaluation reports from third-sector organizations. Assessing the impact of social intervention is becoming increasingly essential for these organizations and public entities that develop similar intervention actions. This approach is because funders of social programs and projects, whether public entities, foundations, companies, or individual patrons, increasingly demand transparency and accountability.
According to the same organization (OECD, 2002, p.4), the impact can be defined as “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.”. Impact evaluation demonstrates how projects, programs, or policies achieve their objectives, showing possible changes in participants' behaviors, skills, and living conditions (Rogers, 2014a).
The outcome of a social intervention's impact may condition its continuation or replication. On the one hand, funders will find it more challenging to re-fund interventions that do not seem to show results. On the other hand, less effective intervention strategies and practices may be internally modified or even abandoned. Promoters and funders can also, based on the impact evaluation results, take advantage of the successful cases and be presented as good practices to be replicated in other contexts and by other organizations. For these reasons, impact assessment is fundamental in seeking greater transparency in social investment and greater effectiveness in social and community interventions.
The need to evaluate social programs and projects promoted by non-profit organizations and public bodies has driven the creation of consulting firms and research centers dedicated to this purpose. Despite several methodologies used to conduct an impact evaluation, starting the process by building a solid “Theory of Change” has become a frequent procedure among evaluators (Rogers, 2014a).
The “Theory of Change” is a model that describes how the activities of programs, projects, policies, or even the mission of specific organizations are expected to produce a series of results that contribute to achieving the intended final impacts. This method can be an asset in understanding the intended mechanisms and outcomes of these interventions to show which indicators should be considered to assess potential changes in people (Rogers, 2014b). A key element to consider in impact evaluation is that its purpose is not only to measure and describe the changes that occur but also to seek to understand the role of that program or project in producing the changes. This process is often referred to as causal inference. Several methods for analyzing causal inference benefit from being based on a solid “Theory of Change” (Rogers, 2014a).
However, it is crucial to note that, in general, the approaches adopted in these evaluations provide only the average effects on the group receiving the intervention, failing to consider the individual effects. However, looking only at average effects can provide misguidance to decision-makers (White et al., 2014a).