By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    Promising Benefits of Predictive Analytics in Asset Management
    11 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Learning to Predict Death with Big Data
Share
Notification Show More
Latest News
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence
How Big Data Is Transforming the Maritime Industry
How Big Data Is Transforming the Maritime Industry
Big Data
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Learning to Predict Death with Big Data
Big Data

Learning to Predict Death with Big Data

DrMarsten
Last updated: 2014/06/07 at 7:57 PM
DrMarsten
8 Min Read
Image
SHARE

ImageDeath. Many of the discussions around data revolve around death. These discussions are typically pretty mundane. Why do phone calls die?

ImageDeath. Many of the discussions around data revolve around death. These discussions are typically pretty mundane. Why do phone calls die? Why do banks go out of business? What makes servers and other network machinery fail?  It’s sad, but sometimes servers die, phone calls are lost, and some long-standing banks suddenly come to an end.

When that happens we use data to discover where it all went wrong. More accurately, we try to use data to discover where it all went wrong, poring through customer accounts data, cell tower network logs, profit and loss statements. That’s the post-mortem, a Big Data autopsy.

Typically these are done using logs or selected data sets, where an analyst pulls some segment of the data for analysis. If that data has structure, there are many tools for querying and analyzing it.  For unstructured data, the long strings of numbers and letters that is the language of machines, solutions have begun to emerge that index the data, making it possible for analysts to search for the signals that will lead to the right answer. In each case, analysts and data experts must comb through data for a cause.

More Read

How Big Data Is Transforming the Maritime Industry

How Big Data Is Transforming the Maritime Industry

Utilizing Data to Discover Shortcomings Within Your Business Model
Small Businesses Use Big Data to Offset Risk During Economic Uncertainty
The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
Analytics Changes the Calculus of Business Tax Compliance

Not too dissimilar is the actual autopsy, which is itself a search for cause. Coming to a definitive cause of death can require a knowledgeable physician, but regardless of the skills of the investigator, prior knowledge of the subject can be critical. Some autopsies are straightforward, but we could easily imagine a situation where there are many contributing factors, some combination of toxins and disease that lead to an untimely death. Decades of experience allow a seasoned professional to compare observations with historical data, but without a perfect match, diagnosis becomes exploratory. That’s a near-perfect analogy to the state of data analytics today. We’ve moved beyond asking simple questions that have simple answers. The challenge of analytics is in finding the combination of events that have lead to death.

The big data specialist typically comes in one of two forms. The first is a practicing data expert, from junior analyst to seasoned data scientist. The second is a subject matter expert, a business user or technology expert with deep practical and applicable knowledge of the business. On the rare occasion that an analyst has both the technical skills and the deep knowledge of the data, their employers will bend over backwards to retain them.

Data science has become a “sexy” job in part because of how rare it is to find someone with the right combination of skills and know-how. These individuals can intuit the combinations of factors that might lead to a specific event better than a pure data person and manipulate data more skillfully than a typical business person.   

This search for answers, however well-informed, has always been approached in an ad hoc manner. Skills and prior experience make for better analysis, but queries are subject to the limitations of human knowledge and the frailties of prior beliefs. Looking for an answer in data starts with a hypothesis, but without further insight, the search becomes exploratory.

What this means for practical application is that we have only scratched the surface of data analysis. For each query that we ask there could be hundreds, thousands, or more that go unasked. As our wealth of data continues to grow, the rate at which we can ask questions remains stagnant while the total number of questions we might possibly ask grows exponentially.

The Promise of Big Data

Of all cigarette smokers, rates of lung cancer death are estimated to range from 8-23%. Simple analysis tells us that smoking cigarettes is a risky activity, but the promise of more data is that, beyond the rate of lung cancer death, we can learn what differentiates the 8% from the remaining 92%. Genetic data, combined with the original smokers data, could reveal the genetic triggers that combine with smoking to differentiate additional conspiring causes of lung cancer death. Even then, we’re not likely to understand every factor that leads to a smoker’s death.

Coming to a reliable answer requires even more data. Not masses of data, but more diverse data. Environmental factors could be at play, like working conditions, sun or chemical exposure, rates of exercise, or levels of stress. Without taking into account this range of data, we simply don’t know.

The promise of Big Data is not that we can see, of all the smokers in the world, exactly what percentage of them will fall victim to lung cancer as a result of smoking. Big Data’s real promise is that across many different data sets, of all different sizes, we can reveal the multivariate factors that lead to death. This is a concept not limited to health data, but must be applied to almost every kind of data.

The Internet of Everything is another example. It will lead to the creation of exponentially more data, but the only thing currently being discussed is the size and speed of the data. When every device in our homes, cars, etc. is connected, we will have lost our ability to have meaningful and impactful analysis by data autopsy. There will be too much health data and too much sensor data for the search model to continue.

In order to realize the true benefits of data we will have to adopt methods that give us a holistic view across data sets without querying data.

Our opportunity for learning more about how we live and work won’t be measured by the size of the data centers it’s stored in, but by the knowledge that’s created when data is combined and analyzed. Connecting all this data across multiple sources and then being able to extract real, meaningful information should be the focus of every organization looking to discover real benefit from data, regardless of scale. Only then will we be able to identify emerging problems as they happen and act on that information before an autopsy is necessary.

DrMarsten June 7, 2014
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

How Big Data Is Transforming the Maritime Industry
Big Data

How Big Data Is Transforming the Maritime Industry

8 Min Read
utlizing big data for business model
Big Data

Utilizing Data to Discover Shortcomings Within Your Business Model

6 Min Read
big data use in small businesses
Big Data

Small Businesses Use Big Data to Offset Risk During Economic Uncertainty

7 Min Read
data-driven approach in healthcare
Analytics

The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive
AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?