Article Preview
Top1. Introduction
In recent years storing data has become both technologically and economically more practical. Data has been flowing rapidly into storage systems and has been accumulating in the last two decades at faster rates than ever. Existing data is more than doubling every two years. The data growth rate has increased from terabytes to petabytes and now to exabytes.
While stored data is extremely large and still growing exponentially, very few effective and efficient tools were created to analyze and find useful information in this data. Businesses, such as supermarkets, typically and routinely collect data through loyalty cards, and always want to know the purchasing behavior of their customers to market their products more efficiently and effectively.
Nowadays, the competitive pressure is very high in some industries. For companies in these industries, it is no longer optional to use data mining techniques since many competitors have been using them already. Therefore, to stay in business and stay competitive, all the companies within the same market segment have to take a look into advances in these technologies, adopt them and use them accordingly.
Businesses now use every means to provide better and more customized services and cutting-edge technologies for their customers or clients. Data mining is one of the major methods to support and customize these services and marketing strategies. The ones which cannot adapt or cannot take advantage of ever-evolving technologies, can lose their customer base very quickly to their competitors.
Nowadays, customers in general are very sophisticated. They know what a good product is, what its fair price is, what brands, companies, services, and products are good or not. All this information is conveniently available to anyone who has access to a computer and internet and has the basic skills of using one of the search engines. Hence, to satisfy their needs a simple query such as the ones created with SQL or google search –which simply gets the subset of stored facts and information from databases, is usually not valuable enough for many users, because they can get this information themselves. Therefore, what is valuable for the people is to have access to not obvious, hidden, or secret information which typically resides inside of this mass amount of raw data. Some companies analyze all the stored transactions and the browsing history of their customers in their databases, analyze the relationships, associations, and correlations among data, interpret, deduce new information from these and use it as a service for their customers.
Amazon (company), for example, uses recommender systems (a.k.a. recommendation systems), which use data mining tools and methods as well as machine learning techniques to explore, associate and extract information from very large datasets and then recommends similar products to its customers. The Amazon application looks at the history, current clicks, and site navigation of a customer, and then recommends customers with similar products that other customers looked at or bought. The recommendation system and product suggestions are supported by the ratings given by their customers. Hence, seeing what others bought and how happy they are with their purchases can make a customer comfortable with making a similar purchase very quickly.
Data mining can explore data and extract valuable information which can be potentially very useful for businesses as we have just mentioned in the case of Amazon corp. Data mining has also been used to help increase or decrease inventories and cut business costs in retail businesses as well as used in government data analysis, law enforcement agencies in their handling and dealing with crime.
Data mining is a method for finding interesting and potentially valuable patterns (itemsets) inside large amount of data. It is mainly divided into descriptive and predictive models. Descriptive models look for patterns, relationships, and associations among these patterns with each other and create rules to describe these relationships. One of the descriptive methods is association rule mining which looks for frequency and co-occurrence of itemsets in the datasets. Association rule can be simply stated as A → B, i.e., given A, B also occurs if the required level of support (minsup) and confidence (minconf) is met.
Association rule mining (ARM) is typically decoupled into two separate processes: frequent itemset mining (FIM) and rule generation. FIM is the more challenging part of ARM as it is an exponential problem as explained in Section 2.2. As the number of unique items increases in the database so does the number of candidates generated and need to be examined. Hence, many algorithms have been proposed for searching for frequent itemsets efficiently.