The Misperception of New Technology – Big Data Edition

Big Data is the answer. It will provide the new customer insights to propel your company to the market leader and do so faster, cheaper, and utilize your existing resources. I’m sure this claim smells a bit to you and with good reason. There are adages for a reason and if “it seems to good to be true then it likely is” seems fitting in this case.

It’s easy to leverage buzzwords in our daily conversations, sales pitches, and marketing materials without really understanding what they mean. Some of today’s buzzwords like “Big Data”, “The Cloud”, “Software-as-a-Service” (SaaS), and “machine learning” are commonly used in interchangeable ways. For example if you’re a SaaS based company with traditional hosting in data centers are you cloud-based or not. If you are then what happens if you transition to AWS? Are you more clouded? I joke to show that there are always gaps between what words mean and how they are actually used.

I’m walking this road today because of an article in the Wall Street Journal titled “The Joys and Hype of Software Called Hadoop” ( and what noise I think it is. The primary point of the piece seems to be that Hadoop and many Hadoop based companies like Hortonworks haven’t been able to meet the expectations of the marketplace and while I don’t dispute that because its always easy to wag a finger at technologies I think the article only touches on the bigger point of using the right technology for the right thing in the right way; and that point is the inherently tougher solve in today’s enterprises.

Big data offers many advantages over traditional technologies like databases and data warehouses. First is the ability to horizontally scale which allows a faster, cheaper (both in hardware and licensing) way to expand your capacity than traditional vertically scaling options in SQL-based worlds. Second is the inherent flexibility of how data is represented and modeled allows for the possibility of combining disparate data sources in interesting ways and often in quicker implementations. And this is where the sell is.

When I decided to make a change from my consulting practice in 2011 I made the choice to seek out a position in a traditional corporation. The purpose was clear. While I had worked with large clients in my past I had never worked in a large organization, I always assumed it wasn’t for me but I knew I had a large gap in my understandings of the working world and I wanted to shore that up. Second, I was considering building enterprise software in the future and wanted to understand my future customer. That was a unique and interesting experience I’ll comment on another day but for this topic I’ll say that I saw huge opportunities for data mining that were not being taken advantage of.

Banks by nature have so much data about their customers that could be leveraged including demographics, spending patterns, location-based data, and life events but the bank I worked for wasn’t close to leveraging all this data. Why is that? First, they were having a difficult time with their warehousing strategy and were backlogged on all those initiatives. And whenever a new one came in it had to be reconsidered against the Master Data Management strategy and then normalized for warehouse operations. If that was ever completed there was then writing the queries and trying to have the business understand what the data meant in the normalized fashion so they could explore it. By that time the budget was slashed and moved to some IT project that was required under banking regulations and the warehouse was never in a state to provide insights.

Big Data feels like a solve in these instances. Leadership can cite how the warehouse team doesn’t produce and look forward to the words “faster” being put into practice but that misses the larger point that the organization isn’t able to deliver their projects today with technologies they seem to understand. That leads to this quote from the WSJ piece, “The dirty secret is that a significant majority of big-data projects aren’t producing any valuable, actionable results”.  And to this I’d say why end there? Aren’t there a plethora of statistics showing how IT projects in the enterprise fail?

Even if a company implements a big data solution it still needs to understand that they will likely have a shortage of the right talent. Newer tech isn’t understood as well by those immersed in older tech and if your workforce isn’t readily adaptive in capability and buy-in you’ll have issues. Also, the problem is far greater than technology because even if you have the data in place you still need the right people with the understanding of the business and marketplace to deliver the insights.

Another adage is that there is no silver bullet or panacea for your business. Whether its agile, outsourcing, offshoring, NoSQL, mobile, or any other buzzword the keyword is leadership. Leadership to have a vision which leads to a plan for how to leverage the right tech at the right time to solve the right thing.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s