In this article we will have a brief discussion about Big Data.

Even if you show even a minute amount of interest in the technical aspects of the net and connected devices, you should be well aware of how humongous amounts of data are generated on daily basis and owe their origins to several sources and so it is necessary to have an analytics layer in order to make the best of all of the data that is available to us. Predictive analysis is only becoming more and more relevant which promises to have a significant positive impact on businesses as well as the bottom line.

But the problem with predictive analysis lies in the fact that it is dependent large sets of mathematical computations and is a process that requires large amounts of memory to be present. So when we computationally deal with Big Data, performing mathematical computations on the same becomes even more difficult.

So we end up facing two particular predicaments:

• Optimizing the process of computing for predictive analysis of Big Data in the presence of computational resources that are comparatively limited in scope.

• And figure out ways through which we may deal with huge amounts of data with memory that is limited.

The solution to this particular challenge may be approached in two distinct ways. The Hadoop ecosystem that taps into the power of parallel computing is considered by many to be the best solution that is available especially so if one considers the fact that it is open source.

Most of the practitioners in this field are well aware that Hadoop has its conceptual basis on cluster based parallel computation and the distributed file system of Hadoop. If you intend to run a machine learning algorithm over the cluster of Hadoop you need a thorough knowledge of map-reduce programming and the learning curve is raised to more difficult levels when you are not well acquainted with the intricacies of programming.

In case your computational resources are limited like having only a single PC, when using Hadoop, we will be unable to perform computational tasks on large datasets. So, in such circumstances, we should continue look for another solution. R and MySQL may together form another viable solution.

We will now to overcome the first obstacle that we mentioned above.

In this case through predictive analysis computation we refer to the task of building a model of machine learning on dataset. A machine learning model comprises of varied formulas of mathematics. Let us now, venture into the intricacies of machine learning predictive model and try to secure an understanding of the reason behind the increased computational difficulty of working with larger sets of data.

A predictive model in a basic manifestation is created through the use of techniques of logistic and linear regression. Now, suppose we are in the process of creating a linear regression model, we face the following challenges:

• Data is so large that we are unable to load it in to memory while using R programming.

• Even when we are able to load data in to our memory, the memory that is left is most often insufficient to perform mathematical computations.

Both the above scenarios require a unique solution that will ultimately let us process large data in R and perform calculations on the same data.

Leave a Reply