Data Aggregation & Data Discovery - Part I

A lot of talk has been heard lately about the concept of data lake.  Variously known as, data refinery, data factory etc.  I find it interesting that we now hear logical architectural terms that speak to the concepts and to the purpose of the big data technologies such as Hadoop / HDFS and Apache distributed database technologies such as HBase/ Cassandra.   

This may be indicative of a shift.  What I am not sure of is does this mean that there is a level of maturity that has been achieved by this suite of open source technologies? Or could  this point to the fact that these technologies have practical applications that solve enterprise scale problems? Or does it show that enterprises have realized that they are no longer able to just deal with "structured data" and that a vast majority of information lies in the space of "unstructured content" leaving them no choice but to venture into the realm of big data technologies?  Not really sure!  

The fact remains, when the big name software vendors start getting into the business of marketing big data technologies and call start publishing white papers with cool sounding names then there is something going on!!  I look at the concepts of data lake, data refinary, data factory etc as synonymous terms for what in the information science realm we call data aggregation!  I could be totally off base here and would love to have more of a conceptual / architectural debate on this topic.


I would love to hear from others actively leveraging these technologies as to how they are applying these concepts/ technologies.

thanks.
surekha -

Comments

  1. VishalJune 21, 2015 at 3:51 PM

    Nice post and I would agree with the fact that Hadoop technology stack is coming of age to solve enterprise scale challenges. Scale was never a challenge for Hadoop in terms of volume, but the maturity to handle both batch and real-time streaming data now elevates the game. However, the concept of data lake is much wider and its implementation can span across multiple technology platforms. I recently came across an Enterprise that chose to implement a large part of the data lake on Hadoop technologies but proposes to drive the consumption through Teradata. Here, the data lake was not only an 'analytical store' but a mission critical platform driving regular Business processes and decisions.

    Populating the data lake with all data across the Enterprise, whether structured or unstructured, in my opinion, helps drive multiple use cases as well help Business uses discover their data assets. This is very cool. However, a word of caution - data has a life cyle and anything that is no longer useful from any of the Business processes or analytical reasons, should be duly retired from the data lake as per the organization's data retention policies to prevent the 'data lake' from becoming a 'data swamp' where users struggle to find what is needed.

    I would also like to hear further from others.

    ReplyDelete
  2. VishalJune 21, 2015 at 3:55 PM

    Also, adding to my previous comment, a robust metadata solution is needed to encourage and drive the wide consumption across an Enterprise. Choice of a physical store for the data lake as well as the associated metadata could be a critical factor for the downstream use cases.

    ReplyDelete

Post a Comment

Popular posts from this blog

Issues with Server Virtualization

Virtualization has been considered a great way to maximize utilization of server resources by increasing server density. In addition, this concept is considered to be very key to green IT initiatvies as it reduces the number of compute resources needing to be supported in a data center thus reducing the cooling and power costs. Having said that, there seems to be a performance downside to virtualization. Given that a virtual machine (VM) is yet another layer of indirection the server HW this layer negatively impacts the performance any time there is a need for the application to access IO Channels, SAN disc and other network resources. This is especially so when the application code is either running inside of a JVM or an application server container such as a JEE container, .NET CLR or even a web server servlet container. In all cases, performance profiles of an application seem to vary whether the application is running inside of a virtual layer vs. directly on the physical server. V
Read more

What is the definition of a strategic partnership with a vendor?

In this post I try to outline a few features of what I would consider a strategic partnership with a service provider or a technology vendor.  These are thoughts that I have distilled from years of experience (good and bad) with my vendor partners.  Hence this aspect of doing business cannot be overlook and not the least of these is the "cost" of doing business. Here are a few key attributes, not necessarily in any particular order of preference. 1) the strategic partner relationship has to go beyond a transactional one to that of a more long term one 2) the strategic partner has to have skin in the game - they have to be measured with your success criteria - they can succeed only if you do 3) the strategic partner has to be willing to "share" their technology roadmap with you 4) the strategic partner has to allow you to re-shape their technology roadmap to meet your needs 5) the strategic partner relationship has to include making their test labs, test gear, pr
Read more