A unified data management architecture that enables organizations to store and analyse structured and unstructured data on a single platform, combining the benefits of Data Lakes and Data Warehouses.
Published in Chapter:
Developing a Data Lakehouse for a South African Government-Sector Training Authority: Implementing Quality Control for Incremental Extract-Load-Transform Pipelines in the Ingestion Layer
Priyanka Govender (Durban University of Technology, South Africa),
Nalindren Naicker (Durban University of Technology, South Africa),
Sulaiman Saleem Patel (Durban University of Technology, South Africa), Seena Joseph (Durban University of Technology, South Africa),
Devraj Moonsamy (Durban University of Technology, South Africa),
Ayotuyi Tosin Akinola (Durban University of Technology, South Africa), Lavanya Madamshetty (Durban University of Technology, South Africa), and Thamotharan Prinavin Govender (Durban University of Technology, South Africa)
Copyright: © 2024
|Pages: 28
DOI: 10.4018/978-1-6684-9716-6.ch006
Abstract
The Durban University of Technology is undertaking a project to develop a data lakehouse system for a South African government-sector training authority. This system is considered critical to enhance the monitoring and evaluation capabilities of the training authority and ensure service delivery. Ensuring the quality of data ingested into the lakehouse is critical, as poor data quality deteriorates the efficiency of the lakehouse solution. This chapter studies quality control for ingestion-layer pipelines to propose a data quality framework. Metrics considered for data quality were completeness, accuracy, integrity, correctness, and timeliness. The framework was evaluated by practically applying it to a sample semi-structured dataset to gauge its effectiveness. Recommendations for future work include expanded integration, such as incorporating data from more varied sources and implementing incremental data ingestion triggers.