Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
    pexels pavel danilyuk 8112119
    Data Analytics Is Revolutionizing Medical Credentialing
    8 Min Read
    data and seo
    Maximize SEO Success with Powerful Data Analytics Insights
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Spark vs. Hadoop: Not Enemies, but Sidekicks
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Spark vs. Hadoop: Not Enemies, but Sidekicks
Big Data

Spark vs. Hadoop: Not Enemies, but Sidekicks

kingmesal
kingmesal
6 Min Read
Image
SHARE

ImageWhile you might have been hearing about Apache Spark and all of the things it can do, you might be wondering whatever happened to Hadoop? After all, MapR is still one of the biggest Hadoop distribution providers.

ImageWhile you might have been hearing about Apache Spark and all of the things it can do, you might be wondering whatever happened to Hadoop? After all, MapR is still one of the biggest Hadoop distribution providers. While you might think that Apache Spark might be replacing Hadoop, that’s anything but the case. Spark represents the next step for Hadoop.

Spark Depends on Hadoop

Hadoop was revolutionary when it burst onto the scene in 2005. Google had its IPO and Facebook had just been founded a year earlier. The cloud was still something you only saw in the sky or on a Windows XP desktop.

More Read

big data analytics for mobile apps
How Big Data Analytics Can Create a Billion-Dollar Mobile App UX
Open Source and free data
5 Keys to Successfully Using Big Data
#16: Here’s a thought…
Data Science shows maturity at 2012 Summit.

The original Hadoop MapReduce made it possible for data centers to process large amounts of data quickly and reliably over a large number of rack-mounted commodity servers. This allowed for scaling by adding more servers rather than having to move everything to a new, bigger machine. With Web 2.0 coming onto the scene, web companies simply didn’t have the luxury of waiting to move everything over to new systems.

Hadoop is the platform of choice for big data at companies like eBay and Yahoo; Spark is also becoming the platform of choice for streaming and batch processing big data. Spark still relies on HDFS to handle the actual data storage. This means that far from going away, Hadoop will become even more important in the future.

Altiscale CTO Raymie Stata agrees:

“To position Spark in opposition to Hadoop is like saying that your new electric car is so cool that you won’t need electricity anymore. If anything, electric cars will drive demand for more electricity.”

What Hadoop Offers

Hadoop’s biggest asset is the Hadoop Distributed File System (HDFS). As the name suggests, it’s a file system for distributing large amounts of data across many servers. The reliable storage system serves as a base for distributed processing engines.

What Spark Offers

Hadoop offers a choice of compute engines under YARN, Spark being one of them. In the past, the most popular had been Hadoop MapReduce. Google wrote the white paper on this framework and used it for indexing all the websites on the internet. With millions of new pages arriving online each day, the engine was designed for processing large amounts of data—but it’s been geared toward batch jobs. No one complains that their page doesn’t show up in Google’s search results the second it’s uploaded. Well, except SEO consultants, perhaps.

Spark, on the other hand, adds in-memory batch and stream processing to the mix. It does this primarily through RDDs, or Resilient Distributed Datasets. These represent data in memory, which is a lot faster than accessing it from hard drives.

With Spark, transformations are performed on RDDs which yield another RDD. Some might think that RDDs and their transformations might hurt performance, but that’s not actually true because of the way transformations are evaluated. Spark borrows another concept from functional programming: lazy evaluation. While you can define transformations, Spark won’t actually compute them unless you perform actions that require a result, such as printing the contents of a file or counting words in a document.

Since the RDDs aren’t actually loaded until you ask for them, this gives you a lot of flexibility in processing. You can use Spark Streaming to process data in micro-batches, even plugging your data into the powerful machine learning it offers.

Dynamic programming languages like Python have opened up new ways to program, letting you develop algorithms interactively non-stop instead of the write/compile/test/debug cycle of C, not to mention chasing the inevitable memory management bugs.

Apache Spark adds the flexibility of dynamic programming to big data, letting you develop new applications and explore your data for trends quickly. Business moves fast, and if you want to stay ahead of your competitors, you’re going to have to move even faster. With Spark, you can build on the solid foundation of Hadoop and HDFS while using Spark to save valuable programmer time.

To give an analogy, Windows 10 launched recently, and while it has some new features, in many ways it’s similar to the previous versions—unless you got the “Something Happened” error when trying to install it.

Conclusion

While Spark offers lightning fast batch processing and stream processing for big data, it still has the tried-and-true HDFS for reliability, giving you advanced processing power and reliability in the same package.

For a more in-depth introduction to Spark, read the free interactive eBook: Getting Started with Spark: From Inception to Production, by James A. Scott.


Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

AI and data mining
What the Rise of AI Web Scrapers Means for Data Teams
Artificial Intelligence Big Data Exclusive
power supplies for ATX for data scientists
Why Data Scientists Should Care About SFX Power Supplies
Big Data Exclusive
AI for website optimization
Free Tools to Test Website Accessibility
Artificial Intelligence Exclusive
Generative AI models
Thinking Machines At Work: How Generative AI Models Are Redefining Business Intelligence
Artificial Intelligence Business Intelligence Exclusive Infographic Machine Learning

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

When Federated Search Bites

15 Min Read

What is R?

8 Min Read
home security and big data
Big DataExclusiveSecurity

3 Crucial Ways Smart Data Eliminates Home Security Threats

8 Min Read

Is love for Twitter blind?

4 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?