By: Vikram Shanbhag – Vice President of Data Solutions
Picture this moment – It’s around the Thanksgiving holiday, you have taken time off and decide to rummage through your old stuff stored away in the attic. You come across pictures, old diaries, post-it notes and other scraps of paper – some of them decades old, simply put aside for future review. As you plough through those items you discover a few long lost friends and business connections that you eagerly set aside. You start making calls and actually connect with some and resume your relationships afresh. These are the data nuggets that you cherish and thank yourself for taking the pains to rummage through and not trashing them in the first place all those years ago.
In today’s Big Data jargon, these nuggets of information are part of what is called Dark Data. This term was invented by Gartner and refers to the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Similar to dark matter in physics, Dark Data often comprises most organizations’ universe of information assets. 
The reason for this Dark Data is the difference between the speed at which data is created and stored versus an organization’s intent and ability to analyze that data for information. Examples of this type of Dark Data are many – web analytics, customer feedback forms and surveys. For a number of reasons – fear of losing important information, regulation, compliance – organizations store record amounts of data which they never get to analyze. This is the corporate equivalent of your Data Attic we discussed above. Some estimates state that most organizations analyze only 1% of their data.  Now imagine the potential value of nuggets of gold stored in that unanalyzed data. It could lead you to a greater understanding of your customer base – why they could be taking certain actions with your products that could impact key parameters of customer acquisition, growth, churn, etc.
As Big Data technologies are advancing in their ability (capacity, speed, analysis) to ingest and analyze data in a real-time manner, it is becoming increasingly viable to gain valuable insights into your data that would have been otherwise relegated to your Data Attic. One of the primary goals of Customer Data Platforms (CDP) is this real-time ability to ingest, analyze and convey data to downstream applications and gain vital customer insights that can be acted upon immediately. This would make organizations much more efficient in using their valued customer data.
Today, leading organizations in retail, finance and healthcare are revamping their customer data strategies and moving towards one driven by a well thought out Customer Data Platform. Those who can get there soon will be able to leverage their customer data and significantly improve their customer experience, revenue and profitability. They will be the ones who do away with their equivalent of the Data Attic.
- Dark Data, Gartner
- The big data challenge of transformation for the manufacturing industry