Top Considerations When Working with Big Data

Daily Finance columnist Anders Bylund (@TMFZahrim) made a big deal about Big Data investments this past week and showcased how BI software is the solution for dealing with all the data once you have your “enormous database.”

While this column was directed at companies to invest in, Bylund provided a good example in calling data analytics software “the muscle to wring meaning from it [Big Data].”

However, it’s not as simple as purchasing software when you consider today’s deluge of data, according to a Search Data Management article. The traditional data warehouse is full of data that’s mostly structured, says Forrester analyst Brian Hopkins (@practicingEA). That’s not the case anymore – today’s data is semi-structured at best.

The “extract, transform and load process” that fed reporting and analytics in the past is not effective anymore. Hopkins says that in this model, less than 5 percent of a company’s data is used.

More Read

In it for the long haul: AWS will now truck data to the cloud

IBM Research’s nanoMRI microscope (via IBMLabs)

“Political prediction markets — in which participants buy and sell “contracts”…”

Join the real movers and shakers in Washington!

Last month, IBM CEO Samuel Palmisano advised President-elect…

The bad news about Big Data is that it can be expensive to go with the traditional data warehousing route when data is so plentiful. The good news is that there are alternatives in technology that can help reduce the cost and increase the usefulness of data.

Wayne Eckerson (@weckerson) says that many organizations are looking beyond the traditional data warehouse solutions and into “emerging big-data technologies such as open source Hadoop and MapReduce.” .

Another strategy comes from Richard Winter, president of Winter Corp., a consulting firm focused on data warehousing. He says organizations can use analytics to “gain business insights [from Big Data] that previously would have been difficult to uncover.” His example was a “smart” asthma inhaler that collects data about the user, when it was used and where. Analysis of this data could lead researchers to deeper insights on how allergens environmental factors affect asthma.

Philip Russom (@prussom), research director for data management at TDWI, says focusing on value can help alleviate the paralysis associated with managing data. He recommends focusing in on a “high payback area such as customer behavior.”

Takeaways:

• Analytics software can help you generate value from all that data, but it’s not the end-all solution. You need a good data management plan as well.
• Big Data doesn’t have to cause paralysis if you focus on areas where data can tell a story or bring insight to a business unit.
• Emerging technologies are helping with the speed of processing and analysis and to reduce costs in data management.

Be sure to follow the TDWI World Twitter feed (#TDWI) as  the week long data warehousing conference  in San Diego ends today.