By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    customer experience analytics
    Using Data Analysis to Improve and Verify the Customer Experience and Bad Reviews
    6 Min Read
    data analytics and CRO
    Data Analytics is Crucial for Website CRO
    9 Min Read
    analytics in digital marketing
    The Importance of Analytics in Digital Marketing
    8 Min Read
    benefits of investing in employee data
    6 Ways to Use Data to Improve Employee Productivity
    8 Min Read
    Jira and zendesk usage
    Jira Service Management vs Zendesk: What Are the Differences?
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
Share
Notification Show More
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Software > Hadoop > Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
HadoopMapReduce

Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4

AlexOlesker
Last updated: 2012/01/31 at 4:00 PM
AlexOlesker
4 Min Read
SHARE

Mahout is a machine learning library that will be included in CDH4

Mahout is a machine learning library that will be included in CDH4

At Cloudera Day, Cloudera software engineed Todd Lipcon Delivered a Deep Dive on the Core of Cloudera’s Distribution including Apache Hadoop (CDH), detailing tweaks and planned improvements to the Hadoop core. Just a few days later, some of these planned improvements were implemented when Cloudera released CDH3, Update 3, and more will be made for the upcoming CDH4. Many tweaks need to be configured by the user, so if you want your cluster running optimally be sure to check the documentation provided by Cloudera.

When measuring the efficiency of a cluster, we can look at three different metrics. Speed can mean per-job latency, measured with a stopwatch, or throughput, measured in slot-seconds. Latency and throughput can be at odds. Another metric, perhaps the most important to Hadoop developers, is overhead, which Lipcon defined as effort spent on jobs you don’t care about. All of these metrics can be improved with some simple adjustments. For example, by tweaking the way Linux IO and caching you can decrease latency by roughly 20% while increasing disk utilization and smoothing out CPU usage.

According to Cloudera, these and other improvements are available in CDH3u3, resulting in a 15% to 150% increase in performance depending on the workload. Additions include MapReduce TaskTracker disk failure toleration, HDFS and MapReduce read-ahead and drop-behind for improved performance, HDFS improved block report performance via improved locking and parallel disk scanning, HDFS shortcut local DataNode reads for improved performance, and  HDFS re-use of client-to-DataNode connections for improved performance. Apache HBase has also been updated with Distributed log splitting on RegionServer crash, Atomic bulk load, and HBCK offline META rebuild as well as updates to Apache Oozie and Zookeeper.

CDH4, which will be available in beta shortly, has an even wider array of improvements and additions based on customer demands and industry trends. One of the biggest changes will be the inclusion of a High Availability Namenode so that if the namenode fails you won’t lose the whole cluster or your data. With Hadoop Distributed File System-RAID, HDFS’s data replication factor will drop from 3 to 2.2 times thanks to a Distributed Raid File System. DRFS increases protection against corruption and hence reduces the amount of replication necessary to ensure availability, for significant disk, rack, and power saving when dealing with Big Data.  CDH4 will also include Apache Mahout for machine learning and improved versions of Flume, Sqoop, and Hue. There will also be new HBase diagnostic and repair tools and Kerberos for security.

Lipcon also mentioned the direction of Hadoop R&D at Cloudera. Goals for future distributions include encryption, disaster recovery, metadata storage and management, resource management, and MapReduce alternatives for problems outside the framework.

TAGGED: Cloudera
AlexOlesker January 31, 2012 January 31, 2012
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai can help with nurse burnout
Breakthroughs in AI Are Helping to Prevent Nurse Burnout
Artificial Intelligence Exclusive
AI in marketing
AI Can’t Replace Creativity When Crafting Digital Content
Artificial Intelligence
ai in furniture design
Top 5 AI-Driven Furniture Engineering Design Applications
Artificial Intelligence
data protection regulation
Benefits of Data Management Regulations for Consumers & Businesses
Data Management

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

Cloudera Leads the Way in Europe [VIDEO]

1 Min Read

SnapLogic: Making Big Data Integration as a Service a Hadoop Reality

4 Min Read

Cloudera Day in DC

2 Min Read

Hadoop is an Open Source Revolution: Federal Computer Week Interview

0 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?