Open Source Software (OSS) for Big Data

Open Source Software And Technology For Big Data

In the past, companies had been writing big checks to database corporations such as Oracle, Microsoft and IBM. After 2000, Google started to encounter a problem that the data they collected were so large that no single database vender will be able to store and process their data anymore, and hence the need for Big Data technology evolved as well as Big Data Analytics.

Balihausen (2019) indicated that the percentage of FOSS (Free and Open Source Software) in the average application exceeds the amount of proprietarily applications exceeds the amount of proprietarily licensed code. According to 2019 Open Source Security and Risk Analysis (OSSRA) report published by Synopsys Cybersecurity Research Center (2019), open source represented 60% of the code analyzed in 2018, up from 57% in 2017, and 64% Open Source Software (OSS) was used for financial services, Big Data, artificial intelligence, business intelligence machine learning.

An entire book on the technology of hands-on approaches to Big Data has been published by Bahga and Madisetti (2016). The following are a few of the tools and frameworks for “batch processing” of Big Data.

Several studies have been completed with the “best” Open Source Software for Big Data in specific categories. These include those of Freeman, Garza et al. (2018) that discussed the best Open Source Software for cloud computing, Heller et al. (2018) the best Open Source Software for data storage analytics, Heller and Pointer (2018) the best Open Source Software (OSS) for machine learning, and Freeman, Heller et al. (2018) the best open software for software development.

Riehle (2019) discussed that the needs of open source processes have led to two major tool investigations that have since become an important part of corporate software development: Software forges and distributed version control. A software forge is a website that allow the creation of new projects and provides developers with all of the tools needed for software development. According to Riehle (2019), distributed version control is version control in which one copies the original repository for Big Data and work with your copy that does not need commit rights or permission to start work. Git and Mercurial are the two best-known examples of such software for Big Data.

Harvey (2017b) provides a detailed study of the top 35 open source companies that use Big Data and play a major role in developing and maintaining the Open Source Software that powers today's businesses.

Frampton (2018) published a complete guide to open source Big Data stack that includes components for visualization, resource management, framework queueing, processing, storage monitoring, and resource management that interact with Apache CloudStack. The following Table 1 provides representative Open Source Software for Big Data for each of these components of Big Data Stack as presented by Frampton (2018) with many of which are discussed in more depth below Table 1 and elsewhere in the following part of this chapter.

[Derived using Frampton (2018)]