Data Aggregation & Data Discovery - Part II

Expanding on the context of Data Aggregation, variously called data refinery, data factory or data lake, I would like to analyze if the concept of Data Aggregation is just a theoretical construct or if there is a practical side to this.

My opinion is that Data Aggregation (regardless of how it is referred to) is just a means to an end; an enabler or precursor for Data Discovery.  This is truly a facility to bring together various types of disconnected sources of data that were previously leveraged in very “targeted” use cases.  The idea being to discover new connections or to explore new usage patterns.   These explorations might belong to the realm of identifying proactive growth opportunities or in the domain of preemptive loss prevention.  Data scientists are able to employ statistical algorithms and predictive modeling techniques to see if new patterns emerge or else to see they are able to ferret out alternate connections.  One also can imagine the use of clustering and machine learning techniques to find unknown patterns that could be applied to marketing, operational process, product placement decisions.  None of these would have been possible with dissociated sources of data.  Thus, the truly quantifiable benefit of leveraging a Data Aggregation platform is to bring large disconnected data into one holistic platform and to run traditional statistical modeling techniques for the purpose of Data Discovery.   

Now you might ask as to how the concept of Data Aggregation and the need to create a platform for disparate data sources are connected to Data Discovery and Big Data?  You are wondering what if any options open up.   Traditional infrastructure has always been a real limiting factor for any enterprise that wanted to create a hosting Platform for large data sets.  So most enterprises would be more inclined to host Data Ware Housing Platforms that offered KPIs which were projections of known trends rather than to invest in Platforms that explored the somewhat unreliable potential that lay in the realm of statistical modeling.  Data scientists overcame this by using “sample data sets”, knowing fully well that the results could be skewed or the patterns discovered could be choppy.  This is where Hadoop based Big Data techniques come in handy. 

Now your IT departments can offer cost-effective Hadoop based Platforms for deploying large data sets needed for Data Aggregation.  Linearly scalable commodity-hardware based big Data Aggregation Platforms make it possible for the data scientist to execute their  predictive models / algorithms against representative  data sets, instead of on scaled down sub-sets.  Most of all, Hadoop based Data Aggregation now insures the reliability of business outcomes generated by these models/ algorithms.  The efficacy of the outcomes and the applicability of the predictions ultimately increase the rate of adoption of data science predictive modeling techniques.  The bottom line – businesses gain a competitive edge from the process of Data Discovery deployed against the Data Aggregation constructs.

I look forward to hearing from you to learn what is working for you and if you have been able to realize the benefits that these technologies seem to tout.

Surekha - 

Comments

Post a Comment

Popular posts from this blog

Issues with Server Virtualization

Virtualization has been considered a great way to maximize utilization of server resources by increasing server density. In addition, this concept is considered to be very key to green IT initiatvies as it reduces the number of compute resources needing to be supported in a data center thus reducing the cooling and power costs. Having said that, there seems to be a performance downside to virtualization. Given that a virtual machine (VM) is yet another layer of indirection the server HW this layer negatively impacts the performance any time there is a need for the application to access IO Channels, SAN disc and other network resources. This is especially so when the application code is either running inside of a JVM or an application server container such as a JEE container, .NET CLR or even a web server servlet container. In all cases, performance profiles of an application seem to vary whether the application is running inside of a virtual layer vs. directly on the physical server. V...
Read more

What is the definition of a strategic partnership with a vendor?

In this post I try to outline a few features of what I would consider a strategic partnership with a service provider or a technology vendor.  These are thoughts that I have distilled from years of experience (good and bad) with my vendor partners.  Hence this aspect of doing business cannot be overlook and not the least of these is the "cost" of doing business. Here are a few key attributes, not necessarily in any particular order of preference. 1) the strategic partner relationship has to go beyond a transactional one to that of a more long term one 2) the strategic partner has to have skin in the game - they have to be measured with your success criteria - they can succeed only if you do 3) the strategic partner has to be willing to "share" their technology roadmap with you 4) the strategic partner has to allow you to re-shape their technology roadmap to meet your needs 5) the strategic partner relationship has to include making their test labs, test gear, pr...
Read more

Data Aggregation & Data Discovery - Part I

A lot of talk has been heard lately about the concept of data lake.  Variously known as, data refinery, data factory etc.  I find it interesting that we now hear logical architectural terms that speak to the concepts and to the purpose of the big data technologies such as Hadoop / HDFS and Apache distributed database technologies such as HBase / Cassandra .    This may be indicative of a shift.  What I am not sure of is does this mean that there is a level of maturity that has been achieved by this suite of open source technologies? Or could  this point to the fact that these technologies have practical applications that solve enterprise scale problems? Or does it show that enterprises have realized that they are no longer able to just deal with "structured data" and that a vast majority of information lies in the space of "unstructured content" leaving them no choice but to venture into the realm of big data technologies?  Not really sure! ...
Read more