Article Preview
TopIntroduction
The business intelligence system has evolved since the growing interest in big data, and therefore data warehouses and OLAP (Online Analytical Processing) (Chaudhuri et al. (2011)). New models of data warehousing (DW) have appeared (Figure 1). We can cite, as examples: (1) DW based on relational database management systems (RDBMS), for example Teradata1, Greenplum2; and (2) DW based on MapReduce paradigm (Dean and Ghemawat, 2004) like Hadoop/Hive (Thusoo et al., 2010a).
Inside a big data warehouse system (Hadoop/Hive) there is still a physical independence between the data warehouse structure and data storage, as we have seen in RDBMS. The main difference relies on data storage. In Hadoop/Hive, data is stored following Hadoop Distributed File System (HDFS) method (block files, data replication, etc.). Nonetheless, there is a convergence between Hadoop/Hive and traditional RDBMS in how to retrieve information from the file system with SQL (Structured Query Language). We only specify what information we need, and the system will compute it to obtain the data from the storage unit. In Hadoop, a MapReduce framework is used to retrieve all data from HDFS and compute them to give the query result.
Since the first version of Hadoop/Hive, it has evolved to become more convenient to use for companies and researchers, but there are still several challenges to tackle, in the big data warehouse and OLAP (Cuzzocrea et al., 2013). One of those challenges is the end-user performance in query execution, particularly in case of multiple-query or intensive-query execution. With the constantly increasing amount of data, we are also facing a growing number of users. In this paper, our primary concern is the number of concurrent queries from users. The problem is known, in RDBMS, as multiple query optimization, or MQO (Sellis, 1988).
In this work, we are interested in the query execution in DW that uses MapReduce framework, particularly on multiple-query execution with MapReduce. As far as we know, in MapReduce framework, all queries are processed in parallel with Map and Reduce tasks and distributed on a cluster of computers. To execute a single query, MapReduce evaluates the query and generates the list of Map and Reduce tasks (Figure 3). During multiple-query execution MapReduce evaluates all queries one at a time, producing an execution latency and an enormous resource consumption, especially due to multiple read/write operations on HDFS distributed file system.
More research has occurred in the multiple-query processing in a distributed environment. Researchers have conducted diversified studies in: (1) system and resources tuning (Herodotou (2011), Herodotou (2012), Marcus (2016)), (2) job scheduling (Zaharia (2009), Isard (2009)) and (3) tasks and/or data sharing (Wang, 2013; Nykiel, 2010; Elgandour, 2012). Those works have pointed out that changing the cluster configuration, ordering the job execution or reusing intermediate results can improve in a significant way MapReduce efficiency during multiple-query or job execution.