By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Top Big Data Challenges Revisited
Share
Notification Show More
Latest News
ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Uncategorized > Top Big Data Challenges Revisited
Uncategorized

Top Big Data Challenges Revisited

Timo Elliott
Last updated: 2015/05/26 at 6:55 PM
Timo Elliott
9 Min Read
SHARE

walk through brain deloitte sap colors

Contents
There’s still a lot of technology to learnThe right people are still hard to findThe right business case is keyIntegrating with existing systems is more importantNew business models are the next big opportunityConclusion

We’ve now been wrestling with Enterprise Big Data for a few years. Here’s a summary of our progress so far.

walk through brain deloitte sap colors

We’ve now been wrestling with Enterprise Big Data for a few years. Here’s a summary of our progress so far.

More Read

big data improves

3 Ways Big Data Improves Leadership Within Companies

IT Is Not Analytics. Here’s Why.
Romney Invokes Analytics in Rebuke of Trump
WEF Davos 2016: Top 100 CEO bloggers
In Memoriam: Robin Fray Carey

There’s still a lot of technology to learn

There’s a lot of new technology to master, and it’s changing fast. There are new in-memory platforms like SAP HANA and open source projects like Hadoop and Spark. There are new techniques such as graph databases, text analytics, spatial processing, machine learning, and many more.

The good news is we’re now past the peak of the Hype Curve. In other words, people have had the chance to experiment and carry out pilots. They now have a better idea of what the new technologies can and can’t do.

When people adopt new technology, they tend to do old things in new ways. But now that people have had a closer look, they’re finding new things to do.

For example, there’s a lot less less talk about ripping-and-replacing traditional analytics and more about wrapping-and-renewing: using Big Data to extend and innovate existing solutions.

The right people are still hard to find

Implementing Big Data means getting the right people with the right skills. It’s still hard to find people who really know these systems.

The good news is that companies have been able to turn to existing analytics staff, who typically jump at the opportunity to learn new marketable skills. This is helped by the quantity of free training resources available online, such as open.sap.com and the SAP HANA academy. These courses explain the detailed technical steps to get value out of both in-memory and Hadoop.

There’s also a shortage of people who can get the most out of all this new data. Harvard business review called Data Scientist the sexiest job of the 21st century (it certainly sounds sexier than “actuary,” which is perhaps the closest 20th century equivalent!). Data Scientists have deep analytic and statistical skills combined with knowledge of the business. They are in high demand and they command high salaries.

What’s new is that technology is helping to remove some of the the bottlenecks. For example there are now easier to use, more automated predictive analytics tools that can be deployed by, say, marketing staff looking to optimize campaigns. This frees up overloaded data scientists for the most strategic projects. And companies are starting to offer “data science as a service.” For example, SAP has teams of people with advanced degrees in statistics who work on co-innovation projects with customers.

The right business case is key

One of the biggest challenges is still building the initial business case for Big Data projects in enterprise environments. And as usual there are two different approaches: either cutting costs, or creating new opportunities.

One of my favorite cost-saving examples is a large european airline. They needed to be able to store and query large amounts of detailed historical information for legal discovery reasons. Initially, they would back it up to tape, and reload it to the data warehouse only when there was a court case. But that required a lot of work so they ended up leaving it in the data warehouse even though it was seldom used. By moving that data to a Hadoop platform, they were able to save a lot of money, and the relatively slow query speed wasn’t a concern for this use case.

Another example is Molson Coors in Canada. They moved their BW system to HANA, not to speed up queries (they were already using the in-memory BW accelerator) but because they had increasing analytic demands, yet limited BI resources. Using SAP HANA vastly simplified the system, resulting in less data storage, lower maintenance, and reduced development effort.

The second approach is to go for something high-value that the company has always wanted to do, but it just wasn’t feasible — and a lot of these have to do with improving customer service.

For example, Center Point Energy in Houston used SAP HANA to create a “mind-reading” call center application. It identifies callers by their call ID, then runs a predictive algorithm using two years of customer history to choose among 40 different reasons the customer could be calling — all in less than a second. It then directs them to the most appropriate service, and provides all the information the operators need to handle the call. The result is that customers get better, faster service at a lower cost to the company.

The project wasn’t new — it was something they had wanted to do for a long time, but previous attempts had failed. In order to be feasible, the results had to be available in less than five seconds. Using traditional systems it took over a minute and a half — far too slow to be useable on the call. And precalculating the information didn’t work either — not only was all that processing and storage expensive, customers were often calling about a bill they had tried to pay just five minutes before.

In all these cases, the initial projects generated buy-in from the business users, and the initial investments were then leveraged for other applications at a lower marginal cost.

Integrating with existing systems is more important

Most enterprise Big Data project today are in standalone silos, with limited links to the existing corporate infrastructure. For example, while it’s clear that Hadoop is enterprise ready in that it can be used for valuable projects, it’s also clear that it has to be a lot enterprise-readier. In order to get the full value of these new technologies, it’s vital to connect them to other corporate data. For example, it’s pointless to track your Facebook likes without accounting for your marketing spend and whether they actually have any effect on sales.

Truly integrated Big Data is still at the planning stage in most organizations. Recently, I’ve been in a lot of meetings with customers who are debating their strategic information roadmaps. They’re looking for new best-practice architecture blueprints that combine the new big data systems not only with traditional analytics, but also with their core business applications.

New business models are the next big opportunity

To get the full value of big data, you have to change your business processes, and change management is notoriously difficult. But of course, it’s also the big opportunity. Because you can now measure new things in new ways, you can create new products and services.

For example, Vodafone Netherlands was able to use predictive analytics to figure out the right people to target for discounted roaming during the ski season. And Kaeser Compressor used the data coming their equipment’s sensors to create new online predictive maintenance services.

Conclusion

Big Data has come a long way. The fact that we’re now in the trough of disillusionment  shows we’re one big step closer to the slope of enlightenment and the plateau of productivity.

 

 

Timo Elliott May 26, 2015
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai in automotive industry
AI Is Changing the Automotive Industry Forever
Artificial Intelligence
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

big data improves
Big DataJobsKnowledge ManagementUncategorized

3 Ways Big Data Improves Leadership Within Companies

6 Min Read
Image
Uncategorized

IT Is Not Analytics. Here’s Why.

7 Min Read

Romney Invokes Analytics in Rebuke of Trump

4 Min Read

WEF Davos 2016: Top 100 CEO bloggers

14 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?