By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Truly Distributed Analytics
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Truly Distributed Analytics
Data Mining

Truly Distributed Analytics

TonyBain
Last updated: 2010/10/21 at 8:19 PM
TonyBain
4 Min Read
SHARE

The growth and success of Hadoop is very interesting.  It is emerging as a highly significant technology for the data scientist.  It is a platform that can scale and accommodate data exploration even across some of the largest datasets that exist today.  Yahoo, I’m told, has a 43,000 node Hadoop cluster.  The mind boggles at the volume of data being crunched with this cluster and ones like it.  Hadoop is distributed.  More specifically, it is a distributed system.

The growth and success of Hadoop is very interesting.  It is emerging as a highly significant technology for the data scientist.  It is a platform that can scale and accommodate data exploration even across some of the largest datasets that exist today.  Yahoo, I’m told, has a 43,000 node Hadoop cluster.  The mind boggles at the volume of data being crunched with this cluster and ones like it.  Hadoop is distributed.  More specifically, it is a distributed system.  A cluster of servers acting together to process a sequence of user initiated jobs. 

While the system may be considered distributed, the data being analyzed is, for all intensive purposes, centralized.  The data at the centre of job analysis jobs must be located within your cluster and directly accessible by your local applications.  This means as the volume of data under the microscope grows the size of the analytics platform grows to accommodate the influx of information. 

However as data science expands external data sources are becoming increasingly relevant for data analytics.  External data being data that is related to your business, but not produced within your organization.  Examples of such data may be environmental data (weather), geographic data (maps, places, addresses etc), shipping & delivery data and so on.  External data can provide insight into irregularity and opportunity within your own datasets that, without it, could be overlooked or misunderstood. 

More Read

supply chain analytics

Automotive Industry Uses Analytics To Solve Pressing Supply Chain Issues

How can CIOs Build Business Value with Business Analytics?
Seven Benefits of Using AI to Perform Text Analysis
7 Data Lineage Tool Tips For Preventing Human Error in Data Processing
Data Analytics Plays a Vital Role in Teacher Verification Software

While I spoke about this the other day somewhat in jest, some silly but simple examples may be the discovery that it is beneficial to increase advertising targeting those in their 30-50’s when “The O.C” is on TV or that it is beneficial to boost the advertising of certain novels in regions where it is currently pouring down outside.  These areas for opportunity couldn’t be discovered until your data is combined with externally sourced data (television scheduling, weather etc).

External data at the moment tends to be quite small and discrete so the current approach is to import external data into the local analytics environment.  And organizations such as Infochimps are doing a great job or organizing these external data sets and providing APIs for importing data into whatever localized analytics platform you are running.  However as the important and volume of external data grows I believe the impact of “importing” this data will grow and the volume of external data may become significantly greater than the local data in certain cases.  Also identifying what external data is relevant will become a role of analytics itself.

While it is early days, one project I am very excited about is focused on how analytics can be distributed between systems and even organizations.  Rather than centralizing large sets of data, the analytics jobs themselves span organizations and data centers.  And of course, when doing so, respecting the security and privacy expectations of all parties in the process.

 

TAGGED: analytics
TonyBain October 21, 2010
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

supply chain analytics
Analytics

Automotive Industry Uses Analytics To Solve Pressing Supply Chain Issues

6 Min Read
Analytics

How can CIOs Build Business Value with Business Analytics?

8 Min Read
text analytics
Text Analytics

Seven Benefits of Using AI to Perform Text Analysis

9 Min Read
data lineage tool
Big Data

7 Data Lineage Tool Tips For Preventing Human Error in Data Processing

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?