Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
    pexels pavel danilyuk 8112119
    Data Analytics Is Revolutionizing Medical Credentialing
    8 Min Read
    data and seo
    Maximize SEO Success with Powerful Data Analytics Insights
    8 Min Read
    data analytics for trademark registration
    Optimizing Trademark Registration with Data Analytics
    6 Min Read
    data analytics for finding zip codes
    Unlocking Zip Code Insights with Data Analytics
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: The Journey from Big Data to Big Promise
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Data Management > Best Practices > The Journey from Big Data to Big Promise
AnalyticsBest PracticesBig DataBusiness IntelligenceData ManagementData MiningData QualityITModelingPredictive AnalyticsSocial Media AnalyticsWeb Analytics

The Journey from Big Data to Big Promise

Ling Zhang
Ling Zhang
13 Min Read
big data
SHARE

big dataIt’s well known that big data is usually stated in terms of the three Vs: Volume, Variety and Velocity. The three Vs appropriately sum up the characteristics of big data and convey that big data is heterogeneous, noisy, dynamic, inter-related and not trustworthy.

big dataIt’s well known that big data is usually stated in terms of the three Vs: Volume, Variety and Velocity. The three Vs appropriately sum up the characteristics of big data and convey that big data is heterogeneous, noisy, dynamic, inter-related and not trustworthy. Companies now strive to convert the three Vs into Big Promises. And Big Data’s promise can be summarized by three new descriptive terms: Veracity, Value and Victory.

 1.      Three Vs of Big Promise – Veracity, Value and Victory

Like the three Vs of big data that well describe the characteristics of big data, the volume is based on both variety and velocity; the three Vs of Big Promise also has an internal relationship. The Veracity mined from big data, based on volume and variety, determines the Value of big data. The value determines the Victory when a business appropriately applies in a timely manner. The higher the Veracity mined from raw data, the more valuable the result, the smarter decision a business can make, and the more successful the business will become. All those will lead to big Victory for the business.

More Read

3 Effortless Tactics to Be a Data Science Success in Business
Happy New Year : 2009 Predictions and 2008 Recap
Key Factors in Choosing Data-Driven Digital Signage Solutions
4 trends in how supply chains are using Big Data
Oracle Unveils the BI Appliance Called Exalytics

While much around big data remains hype, many companies are in the fledging stages of drawing value from their big data corpus, and given an army of discussions and opinions around the topic, it’s still hard to find a clear roadmap to arrive at the Big Promise.

 2.      The Journey from Big Data to Big Promise

Here I share my thoughts about the roadmap from the big picture of big data in a grand view, regardless the type of a business. Basically there are three big steps:

 Step 1: Big Data Collection – Gathering Organic Material

Regardless where you are in the journey – it has to start with understanding the nature of the big data defined by three V’s defined though there is voice that put more dimensions into the big data such as value and veracity. However I do not think they are characteristics of big data in raw. Instead I defined them as two characteristics of big data promise.

 Step 2: Big Data Analytics – Gleaning Big Insight

The core technologies are big data platform and big data analytics.  The big data platform provides the power of speedy processing with millions of records per second. It harness an integrated technologies for transforming organic/raw content to designed content like Natural Language process (NLP), Data Cleansing, transformation (ETL) and filtering methods. The goal aims to transform semi-structured or unstructured data into structured format for easier understanding, analysis and visualization.

 Though in the world of analytics, there are many different kinds of analytics terminologies used and referenced like text analytics, social media analytics, customer, social network, business or sentiment analytics, if given deep thoughts on those terminologies, basically analytics can be categorized into three categories functionally, they are Descriptive Analytics, Relationship Analytics, Prescriptive Analytics. The detail is explained as below for each of them.

 1.      Descriptive Analytics

Once organic data are transformed into designed data from data processing phase, the first analytics is descriptive or exploratory. This phase uses simple statistics to get a general understanding about the data such as data properties like dimensions and field types, statistical profile or summaries like number of records, missing values or field value max, min, median, field value distribution, etc. The exploratory analysis provides us with initial knowledge about the raw content without any deeper digging internal relationships. The process can suggest right strategies to perform deeper analysis. The phase can be done on a random sampled dataset with simple tools like excel sheet and visualized with basic chart types like bar chart, pie chart and scatter plot, etc. The characteristics of the descriptive analytics are:

  • Autonomy, the analytics performed is based on individual fields and their values and it’s self-government and independent of other fields without considering any connections between different fields and contents.
  •  Shallow and Straight forward, the result from the analysis is usually shallow basic statistics like the frequencies of word count, the number and percent of employees with a earning about 5k within a certain geographic area.
  • Simple and Easier understanding – As the method to analyze the data is basic statistical profiling without any extra effort involved, so the result is also simple and easier to understand and visualize.

With descriptive analytics, it can reach a general understanding about what happened. It’s like a doctor to find out what happened to a patient, the fact first before he digs out why the patient got the disease.

  2.      Relationship Analytics

This level analytics aims to dig out embedded valuable insight among the big data. Comparing with the descriptive analytics, the analysis is deeper – in order to succeed at this level, it requires ample mining algorithm or methods like advanced statistics, sophisticated machine learning, inter-disciplined studies, meta or scalable algorithms; the process involved is usually also complicated and performance demanding both in speed and volume.

The reason I called the analytics at this level as relationship analytics because, at this deeper level analytics, its primary goal is to find connection among data elements – the connection may be timely based like sequential dependent relationship or geo location based or functional category based like relationship between production and customer purchasing pattern or transaction based like marketing basket analytics.

During this level analysis, the methods used may be as below:

  • Inferential or Association draws insight from data through random processes that are developed with statistical methods. Inferential depends on the right population and randomly sampled. For example, the average children height tends to higher than their parents who are usually lower than average height of adults. For basket analysis, through mining millions of transactions, some of items have the higher probability to be bought together by customers like coffee and coffee mater – creams, etc. some of the conclusions are easier to understand and make common sense, however, the high value comes from the conclusions that are against people’s common sense or wrong assumption.
  • Model based analysis uses pre-developed model based on the known observed data to infer or predict what will happen in the future. Under this category, two sub categories are commonly known, classification and predictive modeling. Usually when the target variable is in different categories and the method is called classification; when it’s numerical or continuous variable, it’s called predictive method. Both methods need a training data set that are well labeled and a test dataset that are drawn from the same population with the training dataset. The analysis has two phases involved, first a model is built with training dataset then evaluated or tested with test dataset for measuring its performance. Once the model is developed, it’s used to predict the future events or target variables based on the independent variables. For example, a linear regression model can be built to predict sales amount based on the factors that affect sales in the last three months then predict the next month sales; a decision tree model can be built to predict whether a specific twitter message is positive or negative, etc. Sometimes classification and predictive methods are overlapped based on the business applications.
  • Segmentation dynamically group data into different clusters based predefined measurement like distance method. The method is different than the classification or predictive method. It does not need training data or test data. For example, an algorithm can be used to dynamically group similar twitter messages into different clusters.

 3.      Prescriptive Analytics

Prescriptive analysis is actually a business decision based on the conclusions or results drawn from relationship analysis. For a given situation, what kinds of best action to take so that we gain the expected result in the future? Suppose a patient go to see a doctor, first the doctor performs descriptive analysis, fact finding phase, to understand what happened to the patient and some relative factors like daily activities and workloads and food nutrition, next the doctor perform relationship analysis to find out what are the possible factors that cause the patient sick, finally the doctor will give prescription to the patient like medicines to take so that the patient can get well.

 Step 3: Reap Big Promise

In order to fully empower a business with insight drawn from analytics – first the veracity of the result has to be verified before it can be deployed into a business application for generating valuable results. The main approaches that are used to evaluate the veracity of analytics results or models built include precision, recall and accuracy. Also we need to consider the business cost for each error made in dollars. Basically, there are three phases in evaluating the performance, 1) once the model or algorithm developed, the performance can be evaluated based on an validation dataset that are drawn from the same population of the training dataset. If the result is not good enough, the model needs to be redeveloped by adding more data or perform some tuning by adjusting parameters or exploring other methods; 2) the model is evaluated against a test dataset that are drawn from a different dataset than training data. This dataset is more representative to the real world dataset at the point of developing the model and the associate error cost should be also measured based on business objectives; 3) the model will be evaluated on an on-going process. Because the world changes so fast, new data comes in and they may be pretty different from the dataset used to develop the model. The phase 3 should be performed in a regular scheduled base so that the prediction will not go too far off the expected and causes business crashes. In the process, once it’s found the model does not perform well enough anymore, the process will go back to 2).

The values from data veracity of 2) are also count on how well the business takes full advantage of them – how many opportunities to use them to provide business intelligence to customers. Exploring the right business opportunities and defining the right objectives are the key factors for generating business values. If a company can generate higher revenue, victory will be shining out brightly tomorrow.

(image: Big Data journey / shutterstock)

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

accountant using ai
AI Improves Integrity in Corporate Accounting
Exclusive
ai and law enforcement
Forensic AI Technology is Doing Wonders for Law Enforcement
Artificial Intelligence Exclusive
langgraph and genai
LangGraph Orchestrator Agents: Streamlining AI Workflow Automation
Artificial Intelligence Exclusive
ai fitness app
Will AI Replace Personal Trainers? A Data-Driven Look at the Future of Fitness Careers
Artificial Intelligence Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Escaping the ghosts of BI marketing past

1 Min Read
data optimization solutions
Best PracticesBig DataData ManagementExclusive

The Significance Of Data Optimization Solutions For Modern Enterprises

6 Min Read

US Economic Census Treemap

4 Min Read

The Fight for an Internet for the People [VIDEO]

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?