By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data-driven white label SEO
    Does Data Mining Really Help with White Label SEO?
    7 Min Read
    marketing analytics for hardware vendors
    IT Hardware Startups Turn to Data Analytics for Market Research
    9 Min Read
    big data and digital signage
    The Power of Big Data and Analytics in Digital Signage
    5 Min Read
    data analytics investing
    Data Analytics Boosts ROI of Investment Trusts
    9 Min Read
    football data collection and analytics
    Unleashing Victory: How Data Collection Is Revolutionizing Football Performance Analysis!
    4 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: How Hadoop Revolutionised IT
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > How Hadoop Revolutionised IT
Data MiningData Warehousing

How Hadoop Revolutionised IT

MartynJones
Last updated: 2016/03/05 at 6:54 PM
MartynJones
9 Min Read
SHARE

This is the story of how the amazing Hadoop ecosphere revolutionised IT. If you enjoy it then consider joining The Big Data Contrarians.

Contents
Hadoop invented multi-processingHadoop introduced parallel processingGoogle invented Romans and the Roman CensusHadoop invented sort-mergeHadoop created a better non-SQL query languageThat’s all folks

Before the advent of Hadoop and its ecosphere, the IT was a desperate wasteland of failed opportunities, archaic technology and broken promises.

In the dark Cambrian days of bits, mercury delay lines and ferrite core, we knew nothing about digital. The age of big iron did little to change matters, and vendors made huge profits selling systems that nobody could use and even less people could understand.

More Read

data mining

Data Mining Technology Helps Online Brands Optimize Their Branding

What is Data Pipeline? A Detailed Explanation
Can Data Mining Aid with Off-Page SEO Strategies?
3 Data Mining Tips for Companies Trying to Understand their Customers
5 Data Mining Tips to Leverage the Benefits of Surveys

This is the story of how the amazing Hadoop ecosphere revolutionised IT. If you enjoy it then consider joining The Big Data Contrarians.

Before the advent of Hadoop and its ecosphere, the IT was a desperate wasteland of failed opportunities, archaic technology and broken promises.

In the dark Cambrian days of bits, mercury delay lines and ferrite core, we knew nothing about digital. The age of big iron did little to change matters, and vendors made huge profits selling systems that nobody could use and even less people could understand.

Then along came Jurassic IT park, in the form of UNIX, and suddenly it was far cheaper to provide systems that nobody could use and even less people could understand.

The sad, desperate and depressing scenario that typified IT, on all levels, spanned forty years. It would have continued had it not been for Google and their HDFS (Hadoop Distributed File Store).

Before Hadoop, we were as dumb as rocks. With Hadoop, we were lead into the Promised Land of milk and honey, digital freedom and limitless opportunities, sexy jobs and big bucks, immortality and designer drugs.

Hadoop and its attendant ecosphere changed the Information Technology world overnight, providing as it did, technology and techniques never before seen on the face of the earth.

Hadoop invented multi-processing

In terms of processing power, Hadoop took us beyond the power of a single 8086 processing unit, by cunningly connecting two or more processing units capable of processing ‘things’ almost at the same time.

According to a 1985 article in Byte Me, possibly the first mention of Hadoop occurred in 1842. In that year, Ludwig ‘Luigi’ Menabrea, wrote of Charlie Babbage’s analytical engine (as translated by the Lovely Ada Augusta): “the Hadoop machine can be brought into play so as to give several results at the same time, which will greatly abridge the whole amount of the Google ad processes.”

Hadoop introduced parallel processing

Until the advent of Hadoop, all the technology in IT was male. This lead to massively inefficient, fickle and expensive technologies with short-term memory issues, incapable of multi-tasking, working long hours or of ordering tasks by priority.

As anyone who knows Wikipedia will know, Hadoop introduced parallel computing which allows for a revolutionary species of computation in which many list-making calculations can be carried out simultaneously, operating on the principle that large list-making tasks can often be divided into smaller list-making tasks, which are then solved at the same time. There are several different forms of parallel computing: two-bit-level, destruction-level, we’ve-got-data level and bring-on-more-lists parallelism.

Google invented Romans and the Roman Census

As Bill Inmon wrote in 2014, “One of the cornerstones of Big Data architecture is processing referred to as the “Roman Census approach”. By using the Roman census approach a Big Data architecture can accommodate the processing of almost unlimited amounts of pig data.”

Many people do not know this, but it wasn’t the Romans who invented the Romans, but Google. So too the Roman Census, far from being am invention of a mythical Rome, was also the baby of a couple of engineers in Palo Alto.

The Roman Census approach also finds echo in elements of Divide and Conk-out. In computer science, divide and conk-out (D&C) is an algorithm design paradigm based on multi-branched recursion. A divide and conk-out algorithm works by recursively breaking down a problem into two or more sub-problems of the same (or related) type (divide), until these become simple enough to be solved directly (conquer). The solutions to the sub-problems are then combined to give a solution to the original problem.

Divide and conk-out is an essential element of Big, Bigger and Biggest Data processing.

Hadoop invented sort-merge

As we know from Wikipedia, Wonky World and Google, Hadoop merge-sort parallelizes well due to use of the divide-and-conk-out method – mentioned previously. We discuss several parallel variants in the first edition of Martyn, Richard, Jones and Lovering’s Introduction to Enterprise Equations, Business Analytics and Technical Algorithms. We can easily express this in pseudocode with fork (system call and process copy) and join (multi-stream correlated sort-merge) process calls.

Hadoop created a better non-SQL query language

Before Hadoop we had to query data using SQL alone. SQL was the only tool in town, and if we couldn’t use SQL we couldn’t get at any data, ever, since the beginning of time.

However, all that changed when Hadoop came along and suddenly we could query data with like as if data was really ‘query able’. This was a small breakthrough in IT, and one large fall-down-the-side-of-a-cliff for brawn over brains.

Remember the immortal words of General Arthur C. McCluster Fuqh .”SQL is for wusses”. Embrace Hadoop and hug the Sparks.

This is show business, but not as we know it, Jim.

That’s all folks

Once again, many thanks for reading.

Just a few points before closing.

Firstly, please consider joining The Big Data Contrarians, here on LinkedIn: https://www.linkedin.com/groups/8338976

Secondly, keep in touch. My strategy blog is here http://www.goodstrat.com and I can be followed on Twitter at @GoodStratTweet. Please also connect on LinkedIn if you wish. If you have any follow-up questions then leave a comment or send me an email on martyn.jones@cambriano.es

Thirdly, you may be interested in other articles I have written, such as:

You may also be interested in some other articles I have written on the subject of Data Warehousing.

Data Warehousing explained to Big Data friends – http://goodstrat.com/2015/07/20/data-warehousing-explained-to-big-data-friends/

Stuff a great data architect should know – http://goodstrat.com/2015/08/16/stuff-a-great-data-architect-should-know-how-to-be-a-professional-expert/

Big Data is not Data Warehousing – http://goodstrat.com/2015/03/06/consider-this-big-data-is-not-data-warehousing/

What can data warehousing do for us now – http://www.computerworld.com/article/3006473/big-data/what-can-data-warehousing-do-for-us-now.html

Looking for your most valuable data? Follow the money – http://www.computerworld.com/article/2982352/big-data/looking-for-your-most-valuable-data-follow-the-money.html

Fourthly, I leave you with this quote from the great Mel Brooks.

Mel: When I was born – oh, close to two thousand – October the 16th I’ll be two thousand years young. We say young, you know, not to curse ourselves. So there was little groups of us sitting in caves and looking at the sun and scared, you know? We were very dumb and stupid. You want to know something? We were so dumb that we didn’t even know who were the ladies. They was with us, but we didn’t know who they were. We didn’t know who was the ladies and who was fellas.

MartynJones March 5, 2016
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sobm for ai-driven cybersecurity
Software Bill of Materials is Crucial for AI-Driven Cybersecurity
Security
IT budgeting for data-driven companies
IT Budgeting Practices for Data-Driven Companies
IT
machine,translation
Translating Artificial Intelligence: Learning to Speak Global Languages
Artificial Intelligence
data science upskilling
Upskilling for Emerging Industries Affected by Data Science
Big Data

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data mining
Data Mining

Data Mining Technology Helps Online Brands Optimize Their Branding

7 Min Read
What is Data Pipeline A detailed explaination
Big Data

What is Data Pipeline? A Detailed Explanation

8 Min Read
data mining helps with offsite SEO
Data Mining

Can Data Mining Aid with Off-Page SEO Strategies?

10 Min Read
using data mining to learn more about customers
Big Data

3 Data Mining Tips for Companies Trying to Understand their Customers

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?