How to Turn Your Data from Archenemy to Ally

Author: Linda Rosencrance
Spotfire Blogging Team

Author: Linda Rosencrance
Spotfire Blogging Team

Last week we had some fun comparing a data analyst to a superhero. But we all know every superhero has an archenemy – Batman has The Joker and Superman has Lex Luthor.

As a data analyst, you’ll have to confront your share of villains too. They may even have terrifying names like Big Data, Dirty Data or Data Chaos. OK, maybe these aren’t such villainous names but left unchecked they can cause major problems for data analysts and company executives alike.

Why?

Well, for one thing, big data poses big challenges for traditional analytics approaches, according to SearchBusinessAnalytics.com.

That’s because while your company’s focused on the volume and variety of “big data,” it doesn’t really spend enough time figuring out how to transform the massive amounts of stored, raw, structured and unstructured data into useful, real-time business intelligence to make better decisions. 

Then there’s your archenemy, dirty data, which can be a huge headache. Dirty data is data that contains numerous errors like misspellings, duplicate data, data entered in the wrong fields. Dirty data in your system can result in slower performance, reports that are incorrect, and it can cause your software to crash or freeze.

And let’s not forget data chaos. If you want to make the best business decisions, you have to be sure the data is consistent from system to system and from business unit to business unit.

Now that you know who your enemies are, like any self-respecting superhero, you have to figure out how to defeat them and turn potentially disastrous situations into opportunities to make your business more successful.

When it comes to dealing with the problems posed by big data, Jai Vijayan, a former colleague at Computerworld, points to an upcoming report from The Data Warehousing Institute (TDWI).

Vijayan says, according to the survey,  the fastest growing use case for big data analytics is advanced data visualization. Increasingly, companies are running sophisticated analytics tools on big data sets in order to build highly complex visual representations of their data.

And what about handling the problems caused by dirty data?

There are a number of ETL (extract, transform, load) tools that you can use to eliminate inaccurate information from your database(s). There are also some other things you can do to improve the quality and usability of your data including deciding who has the final authority for data hygiene and/or is able to resolve conflicts over whose information is correct, according to this Spotfire blog post.

Finally, you should deploy data governance tools to help you deal with data chaos and ensure the accurate and timely aggregation and reporting of financial results to meet your company’s needs as well as regulatory requirements, according to this article at SearchBusinessIntelligence.com.

One of the things a data governance tool should include is a rollback capability. A data governance tool should enable an application that’s running to revert to the most recent saved version by specifying the transaction name in the ROLLBACK statement.

“A well-generated data system must have rollback capability so that a system can recover to a known state in case the execution process fails,” according to the article.

Remember, as a data analyst, it’s imperative that you get to know these three archenemies and then understand the strategies and best practices needed to convert them into your allies.