The process of collecting and importing information from a variety of sources into an archive, e.g. the Data Lakehouse, so that it can be made available for analysis.
Published in Chapter:
Creating a Data Lakehouse for a South African Government-Sector Learning Control Enforcing Quality Control for Incremental Extract-Load-Transform Pipe
Dharmesh Dhabliya (Vishwakarma Institute of Information Technology, India), Vivek Veeraiah (Sri Siddharth Institute of Technology, Sri Siddhartha Academy of Higher Education, India),
Sukhvinder Singh Dari (Symbiosis Law School, Symbiosis International University, India), Jambi Ratna Raja Kumar (Genba Sopanrao Moze College of Engineering, India), Ritika Dhabliya (ResearcherConnect, India),
Sabyasachi Pramanik (Haldia Institute of Technology, India), and
Ankur Gupta (Vaish College of Engineering, India)
Copyright: © 2024
|Pages: 22
DOI: 10.4018/979-8-3693-1582-8.ch004
Abstract
The Durban University of Technology is now engaged in a project to create a data lake house system for a Training Authority in the South African Government sector. This system is crucial for improving the monitoring and evaluation capacities of the training authority and ensuring efficient service delivery. Ensuring the high quality of data being fed into the lakehouse is crucial, since low data quality negatively impacts the effectiveness of the lakehouse system. This chapter examines quality control methods for ingestion-layer pipelines in order to present a framework for ensuring data quality. The metrics taken into account for assessing data quality were completeness, accuracy, integrity, correctness, and timeliness. The efficiency of the framework was assessed by effectively implementing it on a sample semi-structured dataset. Suggestions for future development including enhancing by integrating data from a wider range of sources and providing triggers for incremental data intake.