Evaluating the Effectiveness of Recommendation Engines on Customer Experience Across Product Categories

Introduction

Artificial intelligence (AI) was established as a field of research at a conference at Dartmouth College in 1956 (Russell & Norvig, 2016). Since then, AI has greatly impacted day-to-day activities and has become a key technology in business. Among various business fields, Verma et al. (2021) suggest that AI will continue to revolutionize the field of marketing. In fact, we observe many AI-powered tools that significantly impact customers throughout all purchase stages of their customer journeys (He & Zhang, 2023). Tools like chatbots, recommendation engines, and virtual assistance also help companies improve their brand awareness and customer relationships (Rana et al., 2022). Polisetty et al. (2023) also investigated the factors that impact a company’s readiness for AI implementation.

As previous studies have shown, AI tools can increase product sales through e-commerce, and thus, firms have incentives to adopt AI tools for their business; however, the effectiveness of AI tools may depend on specific products and their corresponding categories. In particular, the current literature lacks research on how AI tools affect customers differently when purchasing products from distinctly different categories. This study fills the gap between what is known—the fact that AI tools are effective—and our expectation that the effectiveness of AI tools may depend on product categories. To understand how effective AI tools are for each product category, we conducted a survey to examine the basic statistics and performed descriptive analysis on the collected data using machine learning techniques such as similarity analysis and correlation analysis. The study aims to identify the differences, if any, in the performance of AI tools when used for different product categories. Specifically, we evaluate the effectiveness of recommendation engines when customers purchase items from three different product categories: fashion goods, media content (such as music and videos), and consumer electronics products. We also evaluated customer perceptions when interacting with recommendation engines. This research addresses the following research questions (RQs):

• RQ 1: In which product category do consumers use recommendation engines more?

• RQ 2: In which product category do consumers find recommendation engines more effective?

• RQ 3: How do satisfaction levels change at different stages of the customer journey for different product categories?

• RQ 4: What is the area where AI recommendation engines are less effective and need human support?

The rest of this paper presents a literature review and explains the survey design. It then discusses practical insights and considerations, followed by sections on the survey results and data analysis. Finally, we conclude with a summary of this article and a plan for future research.