How to Think Bigger With Big Data

This post was written by Linda Rosencrance, Spotfire Blogging Team

Boom!

This post was written by Linda Rosencrance, Spotfire Blogging Team

Boom!

Big data is exploding. And as we’ve discussed in recent posts, the amount of digital data grows exponentially each year, big data has the potential to become the next frontier for innovation, competition and profit.

Semil Shah (@semilshah) believes for the most part people don’t fully understand what Big Data is yet. But on the surface, at least, they realize it presents businesses with an opportunity to improve their operations. Why? Well, for one thing, storing the data doesn’t cost very much at all and new technologies are being developed every day to help manage, organize, and secure the data, Shah says. 

But for Shah, the real advancements in big data technologies cover how data is structured and stored; how it’s organized and retrieved; and how underlying mathematics can be written into algorithms so the data can be use to discover entirely new things—that’s the part that intrigues him the most.

Shah sees the potential for new techniques in math and physics to enable computer scientists to create new programs to analyze data—programs not currently available—to lead to new discoveries.

“How could our world change if we better understood the underlying mathematics behind the data?” Shah asks. “If finding insights within data is like finding a needle in a haystack, will the right math-based approaches help us build better magnets to draw out those needles?”

Shah talks about taking advantage of stored data—in the public as well as the private sectors—so mathematicians can sift through it using complex algorithms that might let people discover “new products or services; find new deposits of minerals, oil, or gas buried deep in the ground or remote parts of the ocean bed; target geoengineering tactics high up in the clouds to combat global warming; when used in financial markets, not only to notify us of fraudulent behavior, but also to prevent market movers from profiting during bubbles while the masses get doused after the bubbles pop; or analyze seismic activity to predict earthquake likelihoods and tsunami arrival times.”

In this blog, Mark Wojtasiak (@markwojtasiak) acknowledged that there are “mountains upon mountains of data” stored on hard drives, both on private and public servers and in storage arrays, and those mountains are continually growing. The problem is most of that data is not being made public. The question is why?

Wojtasiak thinks rather than just continuing to collect all that data just in case we need it someday, the government as well as private sector companies should turn that data into a natural resource—a new global currency—that mathematicians and physicists can use to make our world a better place.

Shah understands that “the real game-changing innovation and advancement in big data will only come when we’re able to apply the most cutting-edge mathematics and physical sciences to the biggest problems we collectively face.”

And, he adds, so we can, “use the past to impact the future.”

—-

Photo courtesy of ctctlabs.com