Fuzzy-Based Querying Approach for Multidimensional Big Data Quality Assessment

Introduction

In today’s world with an increase in the amount of data processing and information requirement it is essential to develop strategies to effectively manage and assess the data for essential quality checks. The database forms the basis of day to day decisions taken by the organization. Data obtained from employees need to be periodically updated for effective utilization. In this work, an attempt has been made to assess data quality based on certain measures or parameters like Accuracy, Completeness, Reliability, Usability, etc as discussed by Pradheep et al in (2014). Based on these parameters the data set is queried to assess the effectiveness of attributes like accuracy, usability, reliability, timeliness, etc. The parameters or quality factors such as Accuracy, completeness, etc are further subdivided into minor factors. Accuracy is sub divided into Syntactic and Semantic accuracy as explained by Pradheep et al (2014). The sub factors in turn have a parameter which is a measure and this parameter has an acceptable set of values.

To assess the effectiveness of the big data, a model is constructed for a Knowledge Management System which is a Multi-Dimensional Framework for quality checks. The different attributes are each modeled as a decision tree. The combination of all these form a Decision forest tree. A model for the decision forest tree was proposed by Criminisi et al in (2011). The decision forest tree model was a probabilistic model based on classification and regression analysis. The data under consideration could be textual, video, photographs to form a random forest of decision trees. To analyse this data effectively for information several data mining techniques were proposed. Berendt and Preibusch (2014) have proposed several techniques to extract data from databases based on the choice of attributes. Another technique which is map and reduce technique for large data sets had been proposed by Doulkeridis and Norvag in (2014). A large number of data visualizing techniques have been explained in this regard by Doulkeridis and Norvag (2014), Venkat et al (2011), Gorodov et al (2013), Serban et al (2013), Shamsi et al (2013), Jennex and Olfman (2003), Evans et al (2013) and Banerjee et al (2014).

A data dictionary needs to be available which acts as a repository for storing the data. The data dictionary would contain metadata of the data related to the nature, type, volume, etc. Data integrity is another feature which decides on the reliability of data. Data access should be provided according to the role based privileges. This is done based on access privileges and functional aspects. The access privileges may vary from time to time based on the effectiveness of the queries which are also assessed to ensure minimal processing time and memory. Based on this approach the queries are classified into sets. Based on the decision tree analysis carried out the entire data is partitioned into smaller datasets. The size of the dataset may vary arbitrarily based on data volume and processing speed of the database.