Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Software > Hadoop > Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
HadoopMapReduce

Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4

AlexOlesker
AlexOlesker
4 Min Read
SHARE

Mahout is a machine learning library that will be included in CDH4

Mahout is a machine learning library that will be included in CDH4

At Cloudera Day, Cloudera software engineed Todd Lipcon Delivered a Deep Dive on the Core of Cloudera’s Distribution including Apache Hadoop (CDH), detailing tweaks and planned improvements to the Hadoop core. Just a few days later, some of these planned improvements were implemented when Cloudera released CDH3, Update 3, and more will be made for the upcoming CDH4. Many tweaks need to be configured by the user, so if you want your cluster running optimally be sure to check the documentation provided by Cloudera.

When measuring the efficiency of a cluster, we can look at three different metrics. Speed can mean per-job latency, measured with a stopwatch, or throughput, measured in slot-seconds. Latency and throughput can be at odds. Another metric, perhaps the most important to Hadoop developers, is overhead, which Lipcon defined as effort spent on jobs you don’t care about. All of these metrics can be improved with some simple adjustments. For example, by tweaking the way Linux IO and caching you can decrease latency by roughly 20% while increasing disk utilization and smoothing out CPU usage.

According to Cloudera, these and other improvements are available in CDH3u3, resulting in a 15% to 150% increase in performance depending on the workload. Additions include MapReduce TaskTracker disk failure toleration, HDFS and MapReduce read-ahead and drop-behind for improved performance, HDFS improved block report performance via improved locking and parallel disk scanning, HDFS shortcut local DataNode reads for improved performance, and  HDFS re-use of client-to-DataNode connections for improved performance. Apache HBase has also been updated with Distributed log splitting on RegionServer crash, Atomic bulk load, and HBCK offline META rebuild as well as updates to Apache Oozie and Zookeeper.

CDH4, which will be available in beta shortly, has an even wider array of improvements and additions based on customer demands and industry trends. One of the biggest changes will be the inclusion of a High Availability Namenode so that if the namenode fails you won’t lose the whole cluster or your data. With Hadoop Distributed File System-RAID, HDFS’s data replication factor will drop from 3 to 2.2 times thanks to a Distributed Raid File System. DRFS increases protection against corruption and hence reduces the amount of replication necessary to ensure availability, for significant disk, rack, and power saving when dealing with Big Data.  CDH4 will also include Apache Mahout for machine learning and improved versions of Flume, Sqoop, and Hue. There will also be new HBase diagnostic and repair tools and Kerberos for security.

Lipcon also mentioned the direction of Hadoop R&D at Cloudera. Goals for future distributions include encryption, disaster recovery, metadata storage and management, resource management, and MapReduce alternatives for problems outside the framework.

TAGGED:Cloudera
Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Cloudera Day in DC

2 Min Read

Cloudera Leads the Way in Europe [VIDEO]

1 Min Read

SnapLogic: Making Big Data Integration as a Service a Hadoop Reality

4 Min Read

Big Data Tip For The New Project Manager: Starting With Apache Hadoop

3 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?