By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Taming Big Data
Share
Notification Show More
Latest News
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Taming Big Data
Big DataIT

Taming Big Data

MartynJones
Last updated: 2015/04/25 at 9:54 PM
MartynJones
5 Min Read
Image
SHARE

ImageSimply stated, the best application of Big Data is in systems and methods that will significantly reduce the data footprint.

Why would we want to reduce the data footprint?

ImageSimply stated, the best application of Big Data is in systems and methods that will significantly reduce the data footprint.

Why would we want to reduce the data footprint?

More Read

data security in big data age

6 Reasons to Boost Data Security Plan in the Age of Big Data

Growing Demand for Data Science & Data Analyst Roles
How Big Data Is Transforming the Maritime Industry
Top Tools for Your Cloud Data Security Stack in 2023
Boosting Your Chances for Landing a Job as a Data Scientist
  • Years of knowledge and experience in information management strongly suggests that more data does not necessarily lead to better data.
  • The more data there is to generate, move and manage, the greater the development and administrative overheads.
  • The more data we generate, store, replicate, move and transform, the bigger the data, energy and carbon footprints will become.

How can Big Data reduce Big Data?

  • We can use it in profiling, in order to identify the data that could be useful.
  • We can use it to identify immaterial, surplus and redundant data.
  • By using it to catalogue, categorise and classify certain high-volume data sources.

What can we do with the Big Data profile data?

  • We can use it to audit, analyse and review the generation, storage and transmission of data.
  • We can use the data to parameterise data generators and filters, and
  • To be used to generate ‘Big-Data-by-exception’ discrimination rules and as the basis for data discrimination based on directed machine-learning approaches.

So why would we do all of this?

  • We hear that Big Data represents a significant challenge.
  • The best way of dealing with significant challenges is to manufacture an appropriate, coherent and realisable response – a strategy.
  • By addressing the data problems up-stream we can then attempt to turn the Big Data problem into a more manageable data problem, or alternatively, we can choose to remove the problem.

How does this work in practice?

  • We can reduce the amount of data that we actually generate by removing unnecessary generation, storage and transmission of superfluous data. We can change logging, monitoring and signal data generators (applications and devices) so that they produce only concise and usable data. This requires modifications to parts of existing applications and application servers.
  • We can introduce data governors as intelligent data filters and actively exclude or include data in data flows. This is particularly relevant where we are dealing with really high-volume data throughput and bandwidth where release of data into the data streams is subject to rules of exception. For example, we may decide to exclude any market signal data that simply repeats the same price stated in previous data.
  • We can also filter data dimensionally; by association and abstraction of discrete phases, events, facets and values; and, by time, affinity and proximity.

What are the benefits?

  • Making data smaller reduces the data footprint – lower cost, less operational complexity and greater focus.
  • The earlier you filter data the smaller the data footprint is – lower costs, less operational complexity and greater focus.
  • A smaller data footprint accelerates the processing of the data that does have potential business value – lower cost, higher value, less complexity and best focus.

In order to tame Big Data?

  • We should only generate data that is required, that has value, and that has a business purpose – whether management oriented, business oriented or technical in nature.
  • We should filter Big Data, early and often.
  • We should store, transmit and analyse Big Data only when there is a real business imperative that prompts us to do so.

Conclusions?

  • Taming Big Data is a business, management and technical imperative.
  • The best approach to taming the data avalanche is to ensure there is no data avalanche – this is referred to as moving the problem upstream.
  • The use of smart ‘data governors’ will provide a practical way to control the flow of high volumes of data.

Next steps?

If you are interested in the approach to Big Data mentioned here and in particular want to know more about the definition, architecture and use of ‘data governors’ applied to data, then please leave a comment below.

Many thanks for reading.

MartynJones April 25, 2015
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data security in big data age
Big Data

6 Reasons to Boost Data Security Plan in the Age of Big Data

7 Min Read
data science anayst
Data Science

Growing Demand for Data Science & Data Analyst Roles

6 Min Read
How Big Data Is Transforming the Maritime Industry
Big Data

How Big Data Is Transforming the Maritime Industry

8 Min Read
cloud data security in 2023
Cloud Computing

Top Tools for Your Cloud Data Security Stack in 2023

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?