By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data science anayst
    Growing Demand for Data Science & Data Analyst Roles
    6 Min Read
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Preparing Yourself to Move to Apache Spark
Share
Notification Show More
Latest News
SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Software > Hadoop > Preparing Yourself to Move to Apache Spark
Big DataData ManagementHadoopMapReduceUnstructured Data

Preparing Yourself to Move to Apache Spark

kingmesal
Last updated: 2016/02/29 at 1:00 PM
kingmesal
6 Min Read
Image
SHARE

ImageIT is an industry that’s always moving. While some things will never go out of style, like the Unix development model and knowing how to design algorithms properly, often you have to throw out what seems like a perfectly good tool or development style in search of something new.

Contents
Map and ReducePartitioningCountersSerializationConclusion

ImageIT is an industry that’s always moving. While some things will never go out of style, like the Unix development model and knowing how to design algorithms properly, often you have to throw out what seems like a perfectly good tool or development style in search of something new.

While MapReduce has been the mainstay of Hadoop processing, Apache Spark is now taking the throne as the way to handle distributed computation. The reasons are obvious: Spark is very fast due to its use of Resilient Distributed Datasets, or RDDs, and it has a clean programming model.

More Read

data analytics in sports industry

Here’s How Data Analytics In Sports Is Changing The Game

5 Ways to Utilize Data Analytics to Grow Your Business
5 Best Practices for Extracting, Analyzing, and Visualizing Data
Big Data Analytics in eLearning: Aspects Everyone Should Know
5 Current Trends in Big Data for 2022 and Beyond

If you’ve been using the older MapReduce framework, you might wonder how much you need to learn before you can jump in and start using Spark. Fortunately, it’s easy to convert what you already know from MapReduce to Spark.

Map and Reduce

The biggest difference from MapReduce is how the Map and Reduce options are implemented.

To understand this, you need to know how RDDs, Spark’s main data structure, work. RDDs allow for two kinds of operations on data: transformations and actions.

Transformationsare operations that change or filter operations in some way. Transformations are done nondestructively, keeping track of the original data. In a cluster, these changes can be spread out across different nodes. The master node keeps track of these different versions (called lineages) and can rebuild them on another node if one fails. It’s like having a giant RAID. An actionreturns some kind of result.

Given Hadoop’s history, one of the biggest transformations is map(), while reduce() is an action. As with their MapReduce counterparts, map() applies a function to a list, while reduce() aggregates data into a result. The big difference from MapReduce is that these operations are methods exposed by the RDDs themselves.

The programming model is much clearer under Spark than it is under MapReduce. MapReduce is written in Java, which, while not a bad programming language on its own, can be pretty bureaucratic in the way it makes you declare variables. Spark is written in Scala, a much cleaner functional language similar to Lisp or Haskell. 

A program to count lines might take about ten lines or so to write—but you can do it in a few lines of code in Spark with an example borrowed from the Apache Spark Cheat Sheet:

val word1=fm.map(word=>(word,1))
val wrdCnt=word1.reduceByKey(_+_)

What this does is create a variable with the word count in it. We’ve defined the word count by creating key-value pairs with all of the lines, with a value of one in the first line. The second line uses reduceByKey to add all of the keys together. The “_+_” is a placeholder variable meant to add all of the values by key. It’s worth remembering that in Spark, RDDs operate on strings rather than key/value pairs.

We could have squeezed all of this into one line:

val wrdCnt=lines.map(line => (line.length, 1)).reduceByKey(_ + _)

This shows that Spark operations can easily be chained together.

Partitioning

Even though Spark on its own is very fast, you’ll want to make sure that all of the jobs are partitioned well across all your nodes. RDDs can be repartitioned in a customer manner, including by key, or value or even hashing the values.

Counters

Image

You can use counters to count items, such as the line example we showed you earlier. While this can be done using MapReduce, Spark exposes something called accumulators, which are much more powerful. In addition to integers, you can also count using floating point.

Serialization

 

Spark and MapReduce can translate parallel operations back into serial data structures. In both systems, it’s possible to customize how this happens. In Spark, you have a choice of systems, from the default Java implementation to the Kryo version. All you really have to do is set a variable in the Hadoop configuration file.

Conclusion

If this article has whetted your appetite, you might want to try exploring the Spark documentation resources that are available. When you’re ready for a more advanced playground, this is a great resource to get you started with Spark in your own Sandbox.

TAGGED: big data analytics
kingmesal February 29, 2016
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

SMEs Use AI-Driven Financial Software for Greater Efficiency
Artificial Intelligence
data security in big data age
6 Reasons to Boost Data Security Plan in the Age of Big Data
Big Data
data science anayst
Growing Demand for Data Science & Data Analyst Roles
Data Science
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data analytics in sports industry
Big Data

Here’s How Data Analytics In Sports Is Changing The Game

6 Min Read
big data analytics in business
Analytics

5 Ways to Utilize Data Analytics to Grow Your Business

6 Min Read
big data visualization
Data Visualization

5 Best Practices for Extracting, Analyzing, and Visualizing Data

6 Min Read
big data analytics in elearning
Big Data

Big Data Analytics in eLearning: Aspects Everyone Should Know

8 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?