Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
    pexels pavel danilyuk 8112119
    Data Analytics Is Revolutionizing Medical Credentialing
    8 Min Read
    data and seo
    Maximize SEO Success with Powerful Data Analytics Insights
    8 Min Read
    data analytics for trademark registration
    Optimizing Trademark Registration with Data Analytics
    6 Min Read
    data analytics for finding zip codes
    Unlocking Zip Code Insights with Data Analytics
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Taming Big Data
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Taming Big Data
Big DataIT

Taming Big Data

MartynJones
MartynJones
5 Min Read
Image
SHARE

ImageSimply stated, the best application of Big Data is in systems and methods that will significantly reduce the data footprint.

Why would we want to reduce the data footprint?

ImageSimply stated, the best application of Big Data is in systems and methods that will significantly reduce the data footprint.

Why would we want to reduce the data footprint?

More Read

The Horizon of Data Mining
Interview- Phil Rack
What Will We Call Big Data in 2015?
Data Management: Reaching Into the Cloud
Predictive Analytics in Software: Automation That Scales
  • Years of knowledge and experience in information management strongly suggests that more data does not necessarily lead to better data.
  • The more data there is to generate, move and manage, the greater the development and administrative overheads.
  • The more data we generate, store, replicate, move and transform, the bigger the data, energy and carbon footprints will become.

How can Big Data reduce Big Data?

  • We can use it in profiling, in order to identify the data that could be useful.
  • We can use it to identify immaterial, surplus and redundant data.
  • By using it to catalogue, categorise and classify certain high-volume data sources.

What can we do with the Big Data profile data?

  • We can use it to audit, analyse and review the generation, storage and transmission of data.
  • We can use the data to parameterise data generators and filters, and
  • To be used to generate ‘Big-Data-by-exception’ discrimination rules and as the basis for data discrimination based on directed machine-learning approaches.

So why would we do all of this?

  • We hear that Big Data represents a significant challenge.
  • The best way of dealing with significant challenges is to manufacture an appropriate, coherent and realisable response – a strategy.
  • By addressing the data problems up-stream we can then attempt to turn the Big Data problem into a more manageable data problem, or alternatively, we can choose to remove the problem.

How does this work in practice?

  • We can reduce the amount of data that we actually generate by removing unnecessary generation, storage and transmission of superfluous data. We can change logging, monitoring and signal data generators (applications and devices) so that they produce only concise and usable data. This requires modifications to parts of existing applications and application servers.
  • We can introduce data governors as intelligent data filters and actively exclude or include data in data flows. This is particularly relevant where we are dealing with really high-volume data throughput and bandwidth where release of data into the data streams is subject to rules of exception. For example, we may decide to exclude any market signal data that simply repeats the same price stated in previous data.
  • We can also filter data dimensionally; by association and abstraction of discrete phases, events, facets and values; and, by time, affinity and proximity.

What are the benefits?

  • Making data smaller reduces the data footprint – lower cost, less operational complexity and greater focus.
  • The earlier you filter data the smaller the data footprint is – lower costs, less operational complexity and greater focus.
  • A smaller data footprint accelerates the processing of the data that does have potential business value – lower cost, higher value, less complexity and best focus.

In order to tame Big Data?

  • We should only generate data that is required, that has value, and that has a business purpose – whether management oriented, business oriented or technical in nature.
  • We should filter Big Data, early and often.
  • We should store, transmit and analyse Big Data only when there is a real business imperative that prompts us to do so.

Conclusions?

  • Taming Big Data is a business, management and technical imperative.
  • The best approach to taming the data avalanche is to ensure there is no data avalanche – this is referred to as moving the problem upstream.
  • The use of smart ‘data governors’ will provide a practical way to control the flow of high volumes of data.

Next steps?

If you are interested in the approach to Big Data mentioned here and in particular want to know more about the definition, architecture and use of ‘data governors’ applied to data, then please leave a comment below.

Many thanks for reading.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

langgraph and genai
LangGraph Orchestrator Agents: Streamlining AI Workflow Automation
Artificial Intelligence Exclusive
ai fitness app
Will AI Replace Personal Trainers? A Data-Driven Look at the Future of Fitness Careers
Artificial Intelligence Big Data Exclusive
crypto marketing
How a Crypto Marketing Agency Can Use AI to Create Powerful Native Advertising Strategies
Blockchain Exclusive Marketing
data driven insights
How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
Analytics Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

my DeveloperWorks Goes Social

1 Min Read

The AlloSphere

1 Min Read

What You Need to Know About Data Protection in Virtual Environments

15 Min Read

Essential Measures to Manage Big Data From Security Perspective

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
ai chatbot
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?