Insight Into Big Data Analytics: Challenges, Recent Trends, and Future Prospects

Introduction

Big data analytics is the process of extracting hidden patterns and correlations, consumer behavior and preferences, market trends and decision making, by examining huge data sets coming from various sources such as web log files, social media, satellites and sensors, GPS data, IoT (Internet of Things) enabled devices, etc. When we click on a website, a large data is saved in the form of web log files. Which can be used in recommender services in future transactions. Facebook, Tweeter, Instagram and various other social media are generating very huge data every day in terms of contents, tweets, photos etc. They must be saved for further processing. Sensor can be embedded in machines that senses the inputs from the outer world and provide it to the machine for further analysis. Hence, sensors can generate a large volume of data. There many handheld and IoT enabled devices which generated huge data. To extract meaningful pattern from big data, we need to apply application specific analytical methods. As, enormous data are coming from thousands of sources in structured, unstructured and semi-structured formats, it’s a very challenging task to analyze it.

There are following challenges with respect to data storage, data management, analyzing and processing the big data, scalability, privacy and security. Huge data are coming from thousands of sources in different formats, it is a big challenge to store them in an efficient, unambiguous and scalable form. Big data is in the scale of Exabyte. Big data requires special kind of techniques to handle the data. It is not possible for the traditional tools to process the big data. To process them, we need a cluster of machines that can process the data in parallel. So, we use some big data analytics technologies such as Hadoop, Spark, Pig, Hive, etc., to manage and process the big data.

In big data analytics, we deal with huge amount of data with different format, inconsistent, noisy and incomplete data, which generates following challenges. Do all the data need to be analyzed? Do the stored data suitable for analysis? How to find interesting patterns from such a huge, multi-formed, inconsistent, incomplete, uncertain and noisy data? etc. It is much possible that the approach used for big data analytics provides good results on “small” big data but performance degrades rapidly for comparatively larger datasets. It’s a challenging task to produce high quality of information from huge datasets with minimum time, resources and cost.

In recent years, large numbers of techniques and tools & technologies have been developed to analyze the big data. The techniques used for big data are clustering, classification, machine learning, neural networks, topic modelling, etc. To incorporate these techniques to analyze the big data, we have technologies such as Hadoop, Spark, Cassandra, Pig, Hive, NoSQL, HBase, MapReduce, etc. In future, advanced analytics and visualization techniques will be applied on real time business intelligence. To get high performance, in-memory datasets usage will be accelerated.