By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Big Data: What can an energy company teach us about data science?
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Visualization > Big Data: What can an energy company teach us about data science?
AnalyticsBusiness IntelligenceData VisualizationPredictive Analytics

Big Data: What can an energy company teach us about data science?

Ted Sapountzis
Last updated: 2011/06/22 at 5:46 AM
Ted Sapountzis
7 Min Read
SHARE

Two weeks ago I had the opportunity to attend the Economist’s first Information Conference on big data and the disruption it is about to unleash on the world.

Contents
What is big data?Incredible innovation at the data management layer

Two weeks ago I had the opportunity to attend the Economist’s first Information Conference on big data and the disruption it is about to unleash on the world. The event was the brain child of Kenn Cukier who almost 18 months ago wrote this Economist article on the ‘big data deluge’.  The event brought together some of the best minds on this topic, with speakers ranging from academia, technology, NGOs and consultants.

The one topic that received significant coverage during the event was the emerging role of the ‘data scientist’ – a term apparently coined by Jeff Hammerbacher, the co-founder and Chief Scientist of Cloudera while he was at Facebook. The McKinsey Global Institute recently published a study that forecast that the shortage of these skills could be as high as 190,000 people by 2018 in the US alone. The notion of such a discipline bothered me quite a bit and until now was not able to put my finger on it.  In full disclosure, my educational background includes degrees in Mathematics, Computer Science and Operations Research and I have spent most of my career helping companies deal with data and extract insights so they can make better decisions.  But I am getting ahead of myself…

What is big data?

The definition of big data was, to my surprise, not a controversial topic. Most speakers agreed that big data is both about the quantity and quality of the underlying data, i.e., volume measured in petabytes (1015 bytes or 1M gigabytes or more), and data that does not only include structured but also unstructured (i.e., text, video, social media, etc.) data as well.   You can read Wikipedia’s definition here.

More Read

data analytics in sports industry

Here’s How Data Analytics In Sports Is Changing The Game

Four Strategies For Effective Database Compliance
Data Visualization Boosts Business Scalability with Sales Mapping
What Role Does Big Data Have on the Deep Web?
How IoT Can Be Connected to Business Intelligence

Incredible innovation at the data management layer

The field of big data has seen an explosion of a new alphabet soup over the past few years (ACID, Cassandra, Hadoop, HBase, Hive,  NoSQL, MapReduce, Pig, and many more).  Many early-stage (Cloudera, Kognitio, Netezza,  ParAccel) and established (EMC through its acquisition of Greenplum, Microsoft through its acquisition of DATAllegro, Oracle, SAP, and Teradata through its acquisition of Aster Data) technology companies are innovating at an unprecedented pace to help their customers deal with the big data deluge.

While this innovation at the data management layer is significant, most discussions around the data scientist in the industry today are focused at the predictive analytics / data visualization level of extracting insights from big data, and this is wherein my fundamental disagreement lies:

big data

Field is not new – Extracting insights from data (i.e., predictive analytics) gave birth to Operations Research as an inter-disciplinary field during World War II.  The field has its roots in the 1840s based on the work Charles Babbage did to optimize the UK’s mail system.  During WW II, UK and US scientists across many of the same disciplines people talk about today (mathematics, statistics, sociology and psychology) were brought together to help the Allied Forces optimize their artillery rounds and air / sea networks and decipher the German cryptographic codes.  The field then branched out in the 1960s and 1970s in the telecom and airline industries and has since expanded across most of the business world.  The fundamental mathematical techniques however have changed very little in the past 70 years.

big data

We are all data scientists – Most of the innovation that is taking place at the data visualization layer today is about putting the information at the hands of those able to make the best decisions, i.e., the elusive business user / information worker.  While this may feel self-serving as it allows technology companies to expand their footprint, my many years of working as a ‘data scientist’ have led me to the very same conclusion:

  • The real challenge is about driving adoption: Although this is more relevant in an enterprise context, the challenge is not about squeezing the last drop of potential benefit, but rather ensuring recommendations are adopted.  If there is one thing my many years in the field have taught me is convincing the decision makers to adopt your ideas.  This Microsoft Windows 7 commercial sums it up best.
  • Back-office data geeks do not always know the business challenge: Having been one myself, I can attest to the fact despite how smart we think we are, the knowledge that comes from knowing your business while being able to also act on those insights is priceless.  The image and title of this post refers exactly to this point.  PG&E is my local energy utility company and the data on the graph is my hourly energy consumption based on the smart meter (i.e., big) data they collect from my home.  Who better to make decisions about energy consumption than the consumers themselves?  Do the people appearing in this PG&E commercial look like data scientists to you?

What do you think? Is this short-sighted ‘old-world’ thinking, or the reality that will emerge over the next few years as we move past the hype?

 

TAGGED: bi, big data, business intelligence, data management, Data Science, data visualization, enterprise software, operations research, predictive analytics
Ted Sapountzis June 22, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share
By Ted Sapountzis
Follow:
Ted is an entrepreneurial marketing executive with extensive experience and a passion for building and scaling businesses. He is currently the GVP of Product Marketing at Planview, the leader in portfolio and work management solutions.Ted has spent most of his career in enterprise tech and over the course of 20+ years has worn many hats from building award-winning products to making coffee.

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data analytics in sports industry
Big Data

Here’s How Data Analytics In Sports Is Changing The Game

6 Min Read
database compliance guide
Data Management

Four Strategies For Effective Database Compliance

8 Min Read
data visualization for small business
Data Visualization

Data Visualization Boosts Business Scalability with Sales Mapping

7 Min Read
big data technology has helped improve the state of both the deep web and dark web
Big Data

What Role Does Big Data Have on the Deep Web?

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?