Missing data is a common problem for data clustering quality. Most real-life datasets have missing data, which in turn has some effect on clustering tasks. This chapter investigates the appropriate data treatment methods for varying missing data scarcity distributions including gamma, Gaussian, and beta distributions. The analyzed data imputation methods include mean, hot-deck, regression, k-nearest neighbor, expectation maximization, and multiple imputation. To reveal the proper methods to deal with missing data, data mining tasks such as clustering is utilized for evaluation. With the experimental studies, this chapter identifies the correlation between missing data imputation methods and missing data distributions for clustering tasks. The results of the experiments indicated that expectation maximization and k-nearest neighbor methods provide best results for varying missing data scarcity distributions.
TopIntroduction
Information plays a significant role in business to maintain customer satisfaction. Many business successes are based on the availability of information related to marketing strategy. Such information is quite large, and not easy to manage for business purposes. Businesses apply data mining techniques - the process of discovering and processing knowledge from data stored in databases and data warehouses (Han, Micheline, & Jian. 2006) - to overcome information-related problems, such as using analytics to support financial decisions. According to Cios et al (2005), about 20% of the effort is spent on understanding data and problems related to this data, about 60% on data preparation, and about 20% on data mining and knowledge analysis. Specialists spend a great deal of time on data preparation in order to make appropriate decisions because of crucial data quality problems in these datasets in the form of incomplete (missing), redundant or erroneous information. Those problems decrease the quality of data mining tasks such as clustering tasks. Therefore, the treatment of missing data becomes quite important to improve performance of clustering tasks, while missing/incomplete data is the main reason for lack of data quality.
Basically, incomplete/missing data is the missing value of a record in any given dataset/source. Researchers from different areas such as statistics, computer science, etc., have developed many methods and models to analyze large amounts of data to extract valuable information. As has been mentioned above, the quality of information extracted from a dataset is the main concern in knowledge discovery tasks, which is defined as the process of finding and identifying new patterns in the data. These patterns can include relationships, events or trends. In this process, data mining methods are used to extract and verify new patterns. In principle, two types of knowledge discovery tasks can be defined: description and prediction. In a description task, a system finds patterns in order to present these patterns to the user in an understandable way, while a prediction task is defined as finding patterns to anticipate the future behavior of related objects. This study investigates the ways of increasing the quality in description tasks with a particular focus on clustering tasks.
Data quality creates a variety of problems in clustering which is the task of identifying groups of similar objects within a dataset. It is broadly used in many areas such as database marketing, web analysis, bioinformatics, etc. Missing values reduce the power of the clustering process because clustering algorithms usually have no mechanism to handle missing values. A widely used solution for this problem is to fill in the missing values in a preprocessing step. The main goal of this study is to investigate the correlation between missing data imputation methods and missing data distributions for clustering tasks.