By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    Promising Benefits of Predictive Analytics in Asset Management
    11 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
Share
Notification Show More
Latest News
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence
How Big Data Is Transforming the Maritime Industry
How Big Data Is Transforming the Maritime Industry
Big Data
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Software > Hadoop > Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4
HadoopMapReduce

Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4

AlexOlesker
Last updated: 2012/01/31 at 4:00 PM
AlexOlesker
4 Min Read
SHARE

Mahout is a machine learning library that will be included in CDH4

More Read

Cloudera Leads the Way in Europe [VIDEO]

SnapLogic: Making Big Data Integration as a Service a Hadoop Reality
Cloudera Day in DC
Hadoop is an Open Source Revolution: Federal Computer Week Interview
Big Data Tip For The New Project Manager: Starting With Apache Hadoop

Mahout is a machine learning library that will be included in CDH4

At Cloudera Day, Cloudera software engineed Todd Lipcon Delivered a Deep Dive on the Core of Cloudera’s Distribution including Apache Hadoop (CDH), detailing tweaks and planned improvements to the Hadoop core. Just a few days later, some of these planned improvements were implemented when Cloudera released CDH3, Update 3, and more will be made for the upcoming CDH4. Many tweaks need to be configured by the user, so if you want your cluster running optimally be sure to check the documentation provided by Cloudera.

When measuring the efficiency of a cluster, we can look at three different metrics. Speed can mean per-job latency, measured with a stopwatch, or throughput, measured in slot-seconds. Latency and throughput can be at odds. Another metric, perhaps the most important to Hadoop developers, is overhead, which Lipcon defined as effort spent on jobs you don’t care about. All of these metrics can be improved with some simple adjustments. For example, by tweaking the way Linux IO and caching you can decrease latency by roughly 20% while increasing disk utilization and smoothing out CPU usage.

According to Cloudera, these and other improvements are available in CDH3u3, resulting in a 15% to 150% increase in performance depending on the workload. Additions include MapReduce TaskTracker disk failure toleration, HDFS and MapReduce read-ahead and drop-behind for improved performance, HDFS improved block report performance via improved locking and parallel disk scanning, HDFS shortcut local DataNode reads for improved performance, and  HDFS re-use of client-to-DataNode connections for improved performance. Apache HBase has also been updated with Distributed log splitting on RegionServer crash, Atomic bulk load, and HBCK offline META rebuild as well as updates to Apache Oozie and Zookeeper.

CDH4, which will be available in beta shortly, has an even wider array of improvements and additions based on customer demands and industry trends. One of the biggest changes will be the inclusion of a High Availability Namenode so that if the namenode fails you won’t lose the whole cluster or your data. With Hadoop Distributed File System-RAID, HDFS’s data replication factor will drop from 3 to 2.2 times thanks to a Distributed Raid File System. DRFS increases protection against corruption and hence reduces the amount of replication necessary to ensure availability, for significant disk, rack, and power saving when dealing with Big Data.  CDH4 will also include Apache Mahout for machine learning and improved versions of Flume, Sqoop, and Hue. There will also be new HBase diagnostic and repair tools and Kerberos for security.

Lipcon also mentioned the direction of Hadoop R&D at Cloudera. Goals for future distributions include encryption, disaster recovery, metadata storage and management, resource management, and MapReduce alternatives for problems outside the framework.

TAGGED: Cloudera
AlexOlesker January 31, 2012
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

Cloudera Leads the Way in Europe [VIDEO]

1 Min Read

SnapLogic: Making Big Data Integration as a Service a Hadoop Reality

4 Min Read

Cloudera Day in DC

2 Min Read

Hadoop is an Open Source Revolution: Federal Computer Week Interview

0 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?