Big Data and Decision Management Systems: The Impact of Variety

The second in my series on the impact of Big Data on Decision Management Systems: The Impact of Variety

Big Data involves adding more types of data, from more sources inside and outside of the organization, to your analytic toolkit. Social, mobile, local and cloud data sources are exploding and organization must find ways to take advantage of these before their competitors do. This means that the old approach of pulling together all your data into a “360 degree view” simply won’t work any more – you will never get caught up as there is ALWAYS going to be another potentially useful data source. Instead you must begin with the decision in mind and focus on integrating and delivering the data sources you need for a given decision (as Jim Harris and I discuss here).

Variety also has two particular impacts on Decision Management Systems. First it means you have to broaden your definition of data infrastructure. Many (most) Decision Management Systems rely on an operational datastore that is relational and use analytic models built entirely from structured data. With the explosion of new data sources, often unstructured or semi-structured formats, this is not going to work anymore. Your analytic team is going to need to be able to access data stored in a variety of formats (stored on Hadoop for instance) and your operational systems may need to consume less structured records and make decisions against them (what to do with this sensor record, for instance). Same problems (how to build analytics, how to make decisions) but lots of new data sources to deal with.

Second you will need to improve your skills in text analytics and entity analytics. Being able to identify what is being discussed, especially what products or actions, are being discussed in unstructured, text data sources is key. You need to be able to tell that this email is about this product, that this customer keeps talking about the call center etc etc and feed that insight into your modeling and your Decision Management Systems. Being good at analyzing structured data is necessary but, in an era of Big Data, not sufficient.

The final section will handle velocity.

Previous in series