Spark Depends on Hadoop
Hadoop was revolutionary when it burst onto the scene in 2005. Google had its IPO and Facebook had just been founded a year earlier. The cloud was still something you only saw in the sky or on a Windows XP desktop.
The original Hadoop MapReduce made it possible for data centers to process large amounts of data quickly and reliably over a large number of rack-mounted commodity servers. This allowed for scaling by adding more servers rather than having to move everything to a new, bigger machine. With Web 2.0 coming onto the scene, web companies simply didn’t have the luxury of waiting to move everything over to new systems.
Hadoop is the platform of choice for big data at companies like eBay and Yahoo; Spark is also becoming the platform of choice for streaming and batch processing big data. Spark still relies on HDFS to handle the actual data storage. This means that far from going away, Hadoop will become even more important in the future.
Altiscale CTO Raymie Stata agrees:
“To position Spark in opposition to Hadoop is like saying that your new electric car is so cool that you won’t need electricity anymore. If anything, electric cars will drive demand for more electricity.”
What Hadoop Offers
Hadoop’s biggest asset is the Hadoop Distributed File System (HDFS). As the name suggests, it’s a file system for distributing large amounts of data across many servers. The reliable storage system serves as a base for distributed processing engines.
What Spark Offers
Hadoop offers a choice of compute engines under YARN, Spark being one of them. In the past, the most popular had been Hadoop MapReduce. Google wrote the white paper on this framework and used it for indexing all the websites on the internet. With millions of new pages arriving online each day, the engine was designed for processing large amounts of data—but it’s been geared toward batch jobs. No one complains that their page doesn’t show up in Google’s search results the second it’s uploaded. Well, except SEO consultants, perhaps.
Spark, on the other hand, adds in-memory batch and stream processing to the mix. It does this primarily through RDDs, or Resilient Distributed Datasets. These represent data in memory, which is a lot faster than accessing it from hard drives.
With Spark, transformations are performed on RDDs which yield another RDD. Some might think that RDDs and their transformations might hurt performance, but that’s not actually true because of the way transformations are evaluated. Spark borrows another concept from functional programming: lazy evaluation. While you can define transformations, Spark won’t actually compute them unless you perform actions that require a result, such as printing the contents of a file or counting words in a document.
Since the RDDs aren’t actually loaded until you ask for them, this gives you a lot of flexibility in processing. You can use Spark Streaming to process data in micro-batches, even plugging your data into the powerful machine learning it offers.
Dynamic programming languages like Python have opened up new ways to program, letting you develop algorithms interactively non-stop instead of the write/compile/test/debug cycle of C, not to mention chasing the inevitable memory management bugs.
Apache Spark adds the flexibility of dynamic programming to big data, letting you develop new applications and explore your data for trends quickly. Business moves fast, and if you want to stay ahead of your competitors, you’re going to have to move even faster. With Spark, you can build on the solid foundation of Hadoop and HDFS while using Spark to save valuable programmer time.
To give an analogy, Windows 10 launched recently, and while it has some new features, in many ways it’s similar to the previous versions—unless you got the “Something Happened” error when trying to install it.
Conclusion
While Spark offers lightning fast batch processing and stream processing for big data, it still has the tried-and-true HDFS for reliability, giving you advanced processing power and reliability in the same package.
For a more in-depth introduction to Spark, read the free interactive eBook: Getting Started with Spark: From Inception to Production, by James A. Scott.