The process of extracting, transforming and loading the data, namely the extraction of data from its data sources to a staging area, the transformations to add structure or to clean the data and the act of loading it to its final destination.
Published in Chapter:
Real-Time Big Data Warehousing
Francisca Vale Lima (University of Minho, Portugal), Carlos Costa (University of Minho, Portugal), and Maribel Yasmina Santos (University of Minho, Portugal)
Copyright: © 2019
|Pages: 30
DOI: 10.4018/978-1-5225-5516-2.ch002
Abstract
The large volume of data that is constantly being generated leads to the need of extracting useful patterns, trends, or insights from this data, raising the interest in business intelligence and big data analytics. The volume, velocity, and variety of data highlight the need for concepts like real-time big data warehouses (RTBDWs). The lack of guidelines or methodological approaches for implementing these systems requires further research in this recent topic. This chapter presents the proposal of a RTBDW architecture that includes the main components and data flows needed to collect, process, store, and analyze the available data, integrating streaming with batch data and enabling real-time decision making. Using Twitter data, several technologies were evaluated to understand their performance. The obtained results were satisfactory and allowed the identification of a methodological approach that can be followed for the implementation of this type of system.