There goes that universe screaming at me again. Just when I thought I can move on to try out my ideas for the “Paradoxes of Business Intelligence” there were several events all geared towards one thing: Big data
Just a funny tangent, I remember joining Teradata when everyone wondered “who would ever need a terabyte of data”. Today it is hard to find someone that doesn’t have a terabyte of data.
The first shout out to me was the Teradata acquisition of Aster data. Teradata has always been known for scalability and “big data” but this is going to a whole new level. Aster data allows people to take extremely large data sets and run map reduce intelligence to find the small, yet highly relevant, parts you really care about.
Editor’s note: Rob Armstrong is an employee of Teradata. Teradata is a sponsor of The Smart Data Collective.
The second shout out was at a happy hour party. Several people were talking about the hue and cry over Apple capturing location data and using it for analytical purposes. Then people figured out that this was not that big a secret and many other companies do the same thing (every track your cell phone via cell towers?). There was a great blog post by Stephen Baker related to this topic specifically. The consensus was that this would be a very large amount of data and what could they possible do with the information. As I was off the clock, I did not go into all my BI stories (just some of them…)
I say that as long as that information is being captured and used in a way that I get value from it then great. Of course what is valuable to me is different to what is valuable to others, and therein lays the opportunity. We are starting to move from a world of capturing transactions and leading companies are starting to capture interactions in order to make more relevant actions with their customers.
This transition has been happening for several years. I was talking with a telecommunication prospect in India about 5 years ago. I asked if they are able to identify a busy signal. Of course they could do that in the network. I then asked if they captured that event in the data warehouse. I got a very incredulous look from the IT group and they said, “do you know how much data that would be?” My response was that yes I get that is a lot of data but did they understand the business value that information would enable. Luckily we had a business person in the room and they were able to rattle of just a few opportunities (second line and call waiting being the obvious ones).
So how big does this get? Real big, real fast. If you think about it, every transaction you have with a company is the result of at least one interaction. Most likely it is at least 5 to 10 interactions. Then you need to consider that not every interaction (or series of interactions) will result in a transaction. Companies are now implementing systems that are not only 10’s or 100’s of terabytes but even petabytes (1,000 terabytes) in size.
So what can you do with that much data?
If you are a retailer and capturing web browsing you can now see who comes to your store (website), understand in detail what they look at (web pages and links) and then correlate that with what they end up finally purchasing (checkout page). This can help in web and store design, pricing, targeted marketing, and inventory management.
If you are an insurance provider and are able to track cars over time and space then you can target premiums to risk. There are ways to capture a car’s location and discrete intervals (say every minute). You can now understand when people drive on what roads, at what speeds. Target discounts for good drivers and charge more for bad drivers, and do this to an individual rather than a “group” based on age or home address.
There are many more examples throughout every industry. Just think about your situation and what you should know about your customers. Better yet, think about your own life and all the interactions you have with companies and how they could provide you with better service if they only understood you better.
If you are a regular reader of my thoughts, you know this is one of the areas that you had to plan for up front. This is explosive growth and creates a new supply of analytics as well as challenges in the ETL processes. Are you ready for this? If so, can your business users exploit such data? If not, what are the barriers and how are you going to overcome them?
