Demystifying Big Data (Part 2)

In my last blog I spoke about how we can leverage Big Data for various business operations from human resource management to supply chain management and marketing initiatives.Moreover I spoke about how to leverage Big Data tables like Cassandra to enhance the capability of customer relationship management systems. Now moving one step ahead we need to take a look at how much data is “Big Data”. In an era when this new buzzword is  ringing bells aloud in the ears of CIOs/CTOs and other decision makers of organizations big or small, we need to be cooperative enough to make them understand how much of data is “Big Data” for them. This is an  attempt to address this need of the hour.

Firstly, please get this right that not the entire data getting generated on earth through clicks or Social media or other digital processes and getting stored as petabytes of data (this is how it is quoted by Big Data proponents) is not Big Data for you. Big Data advocates are saying that in the last 2 years the amount of data that has got generated is more than it happened ever and that is Big Data and you need to make use of it. This is incorrect. This approach is mystifying Big Data and opportunities which lay hidden for organizations as they are afraid of petabytes and huge resource intensive analysis which they would perhaps need to execute to harness insights from that endless and ever growing repository of unstructured data.

So the million dollar question is “how much data is Big Data for you?”

Say you are a retailer with multiple stores all around with a heavy customer base. You have an RDBMS database with customer dimension table which gives you a limited view of the customer and attributes available in there are already exhausted in terms of marketing opportunities. You are trying to add more dimensions to your customer base and leverage those attributes to create some more innovative marketing campaigns to increase sales. Say here you want to leverage you instore closed circuit camera recordings to analyze which are the aisles where customers are spending more time and which customers are evaluating various products in front of a particular aisle. Now these are digital video data which is in an unstructured format and cannot be used by your marketing department straight away. This dataset needs to be stored in a Big Data table and then can be analyzed down to customer level information. So this is Big Data for you which was out of scope earlier  and not feasible with an RDBMS database.

Now moving on to the next part of this , is how this unstructured data is analysed and converted into an attribute which can be leveraged by your marketing team. In the NoSQL database you can retrieve this information in JSON format which our marketing Automation systems like IBM Unica or Teradata Aprimo or others cannot retrieve to make use of. So you need an adapter to convert this JSON format data to SQL format and add it to an RDBMS database as an attribute. Only then this insight comes to use to a marketer.

A few takeaways from what we have discussed above to understand “Big Data” better:

1. Big Data is bigger than your on-premise data-warehouse but not necessarily peta-bytes of data always.

2. Big Data is that amount of data which you envision is required to add that extra bit of dimension to achieve a certain business goal.

3. Big Data is highly manageable if the requirement analysis and usage mode is pre defined properly.

4. Big Data is an ocean but with a proper navigation system you can easily identify the island where you want to pause.

Hope this helps and if you have any comments please share with me on avik.c@avankia.com