IDG Contributor Network: How to answer the top three objections to a data lake

We’ve all seen the marketing hype surrounding the data lake. Data lakes are much like Michael Corleone at the end of The Godfather. Data lakes will answer all your questions and solve all your problems. However, as with Michael’s pronouncement(s) at the end of The Godfather, there is a downside to this “offer” that marketers may think we cannot refuse. There is usually a set of stakeholders out there who are unfamiliar with Hadoop or the concept of a data lake or perhaps just not interested in changing the status quo of their organizations.

As a data architecture, you are pitching a data lake like you do one of those mountain lakes on travel websites or George Clooney movies … lakes are cool, clear, and usually have the reflection of a snow-tipped mountain peak on their surface to show the purity of the contents within. Everyone wants to drink water from this source. However, when some people hear the concept — data from many sources being stored without a schema for some possible future benefit — they will think more about the concept of a data swamp rather than a pristine data lake.

Data swamps are places where unknown data sits in a Hadoop cluster. You don’t know where the data came from. You don’t know how old the data is. You have no idea what you might use the data for. Heck, the first use of this type of data before a skeptical executive more concerned with the status quo than organizational change will evoke the classic, “what’s your data source? How can you verify this information? I have different experiences….”

But before you can even get to that meeting where people start to question the data from your data lake, you need to propose, build, and populate one. Here are the top three (3) objections that I often hear to “discourage” any budding data architect from attempting start their data lake initiative, and how you might answer those objections:

Aren’t data lakes just another silo to get in the way? Just like the name implies, data lakes provide the opportunity to put all that pure data into a single location. This allows for information from those new, and often voluminous, data sources to share an environment with traditional data sets and each other. This allows for data-driven organizations to discover links between data sets such as mobile and social, make new insights from the data, and potentially create new business models such as how Uber changed the personal transportation business. I would answer this objection with the advances in data integration technologies such as data virtualization and ETL/ELT/ET/ETLT, as well as the ability to share data between data management architectures. The day of “data silos” is more about “want to” than “can’t do.”

Data lakes aren’t robust enough for our needs…Hadoop isn’t even 10 years old! I would say that the above objection is provided by someone who is invested in the care, feeding, and maintenance of a data warehouse. The types of “needs” that this objection is attempting to address are data governance, quality, stewardship, and lineage. True, the data governance practices of data lakes lags behind those other data architectures based on the concept of ‘schema on write’ where you predetermine the questions before you create and populate the structure. I would answer that a data lake attempts to solve a different set of requirements. Instead of assuring the quality of the data for “regulatory quality reporting” (i.e., someone goes to jail if the numbers are wrong), data lakes are designed to allow for discovery and then the potential use for new business models. A data lake’s data quality practices are less about the syntactic quality of the data (are all the fields perfect?) and more about the semantic quality of the data (can we use this well?).

Data lakes threaten the established data management structures such as the data warehouse More often than not, I hear this one coming out the mouth of someone who sells proprietary data warehouse storage components…yes. Some in the EDW world find the presence of the data lake to be a threat to the “single version of the truth” component of the enterprise data warehouse. However, more often than not, the data that exists within a data lake isn’t the type of curated structured data that data warehouses are known for. I would answer that the data that exists within the data lake is more often the type of atomic level event data with lots of extra fields that haven’t proven themselves yet “worthy” of placement in the data warehouse. Part of this is the concept of separating the signal from the noise. Another is the concept that pouring potential petabytes of data into the EDW will cause to two things to happen. One, the data quality people will become “concerned” (okay, have a heart attack) over the data coming into the platform. Two, the storage vendor will retire early to some golf course with the purchase agreement to handle all that information

After you hear the objections a couple hundred of times, the question then becomes: is a data lake worth the time, trouble, and effort if it might devolve from the pure data sources high in the mountains if this is the type of resistance that you encounter? The answer to that is “most certainly!” The advantages of the data lake outweigh the risks. The data lake is how data-driven organizations will validate and power their new businesses.

Does your organization want to be part of the future or part of the past?

This article is published as part of the IDG Contributor Network. Want to Join?

Source: InfoWorld Big Data

IDG Contributor Network: Five core attributes of a streaming data platform

As your data-driven organization considers incorporating new data sources like mobile apps, websites that serve a global audience, or sensor information from the internet of things, technologists will have questions about the required attributes of a streaming data platform.

There are five core attributes that are necessary for the implementation of an integrated streaming platform and allow for both the acquisition of streaming data and the analytics that make streaming applications possible:

Low latency: Streaming data platforms need to match the pace of the data sources that they will acquire data from as part of a stream. One of the keys to streaming data platforms is the ability to match the speed of data acquisition with the requirements of the near real-time analytics needed to disrupt particular business models or markets. The value of real-time streaming analytics diminishes when you have to wait for the data to be landed in a data warehouse or a Hadoop-based data lake architecture. In particular, for location-based services and predictive maintenance applications, the time between when the data is created and landed in a data management environment represents a missed customer opportunity at the least or a stranded multi-million dollar asset critical to your business operations at the most.

Scalable: Streaming data platforms are not just connecting a couple of data sources behind the corporate firewall. Streaming data platforms need to be able to match the projected growth of connected devices and the internet of things. This means that streaming data platforms will need to be able to stream data from a large number of sources — potentially millions or even billions of sources, both internally and externally.

IDG Contributor Network: Introduction to streaming data platforms

Data-driven strategies are becoming a greater part of an organization’s DNA. Executive management is embracing how data can be used to create, sustain, and strengthen competitive advantage. Disruptive companies are building business models based on data that other organizations leave behind. Employees are growing into the role of data analysts as part of their day-to-day responsibilities, and companies are introducing new data sources to continue this trend.

Specifically, data-driven strategies can be seen in the way that organizations are taking advantage of modern technical architectures in mobility, cloud, and device sensors, and integrating that information into new ways of doing business. The use of location-based mobile apps, optimized supply chains for online retail applications, and the introduction of the internet of things, increased the focus on low-latency data collection, transformation, and analytics.

With the rise of importance of data-driven organizations and the focus on low-latency decision making, the speed of analytics increased almost as rapidly as the ability to collect information. This is where the world of streaming data platforms comes into play. These modern data management platforms bring together not just the low-latency analysis of information, but the important aspect of being able to integrate information from operation systems, mobile applications, databases, and the Internet of Things in real-time/near real-time.

The true key to streaming data platforms, and the applications they support, is the integration of technical and business data sources in real time. Without this level of streaming data acquisition, or data integration, the analytics that data-driven strategies and the business models built on those strategies cannot match the promise of the business stakeholders who are looking to create new business value and increase competitive advantage.