Multidimensional Perspective to Data Preprocessing for Model Cognition Verity: Data Preparation and Cleansing - Approaches for Model Optimal Feedback Validation

1. Introduction

In an era where embedded cognition largely affects the strategic decision-making of an organization and by extension a people, there is a dire need for quality assurance in the preparation of the data that governs such systems. As the saying goes, “Garbage-In-Garbage-Out”, unreliable data fed into a reliable model produces an unreliable prediction which leads to unreliable decisions. Data truth (originality) and statistical truth (data significance) inform the business truth. Business Truth is a measure of growth upon which business processes and key performance indicators (KPI) are created and established. Ultimately, the breakdown of most organizations and models is not unconnected to faulty and unverified decision-making because of wrong data.

Zarepoor et al. (2021) affirm this assertion by feeding different machine learning (ML) models with ill-prepared and imbalanced data. This resulted in the underperformance of all the models based on oversampling. The systematic literature survey carried out by Eyuboglu et al. (2022) shows that the systematic errors with cross-modal embeddings are because of poorly prepared data used by the system for data analytics. Furthermore, Li (2021) shows that high generalization of faulty signal data resulted in faulty diagnosis and wrong prediction. Finally, the works by Huang et al. (2021) indicate that the underperformance of a novel 3D-based deep learning model using a meta-learning paradigm was attributed to a misguided step in the construction of the image data used for the model simulation. Hence, the steps undertaken in preparing quality data determine the performance of a model and the reliability of its prediction.

However, developing a robust and reliable dataset (a relatively error-free dataset) is a herculean task that requires professionalism and precision to ensure that quality, and sample inclusiveness are maintained. Due to the costly effect a slight error in data can have on the system, careful steps are taken in phases to ensure precision and near-perfection in dataset preparation. These phases include determining the data collection site, acquisition of right tools for data capturing, method for data generation, deciding on the sampling technique to adopt, labelling/annotation, cleaning, clustering/categorization, etc. Each phase requires domain-specific expertise. For instance, to generate image-based dataset used by deep learning models to detect drones and other unmanned aerial vehicles (UAV) involves interwoven steps undertaken over a long period (Ajakwe et al., 2022a). A wrong choice of data collection equipment for this task leads to wrong data generation and preparation. Figure 1 highlights the tools for generating and preparing image-based UAV data.

Figure 1.

Tools for UAV data generation and preparation

(Ajakwe et al., 2022)