Improving Hadoop Performance with Optimization, CDH3 Update 3, and CDH4

At Cloudera Day, Cloudera software engineed Todd Lipcon Delivered a Deep Dive on the Core of Cloudera’s Distribution including Apache Hadoop (CDH), detailing tweaks and planned improvements to the Hadoop core. Just a few days later, some of these planned improvements were implemented when Cloudera released CDH3, Update 3, and more will be made for the upcoming CDH4. Many tweaks need to be configured by the user, so if you want your cluster running optimally be sure to check the documentation provided by Cloudera.

When measuring the efficiency of a cluster, we can look at three different metrics. Speed can mean per-job latency, measured with a stopwatch, or throughput, measured in slot-seconds. Latency and throughput can be at odds. Another metric, perhaps the most important to Hadoop developers, is overhead, which Lipcon defined as effort spent on jobs you don’t care about. All of these metrics can be improved with some simple adjustments. For example, by tweaking the way Linux IO and caching you can decrease latency by roughly 20% while increasing disk utilization and smoothing out CPU usage.

According to Cloudera, these and other improvements are available in CDH3u3, resulting in a 15% to 150% increase in performance depending on the workload. Additions include MapReduce TaskTracker disk failure toleration, HDFS and MapReduce read-ahead and drop-behind for improved performance, HDFS improved block report performance via improved locking and parallel disk scanning, HDFS shortcut local DataNode reads for improved performance, and  HDFS re-use of client-to-DataNode connections for improved performance. Apache HBase has also been updated with Distributed log splitting on RegionServer crash, Atomic bulk load, and HBCK offline META rebuild as well as updates to Apache Oozie and Zookeeper.

CDH4, which will be available in beta shortly, has an even wider array of improvements and additions based on customer demands and industry trends. One of the biggest changes will be the inclusion of a High Availability Namenode so that if the namenode fails you won’t lose the whole cluster or your data. With Hadoop Distributed File System-RAID, HDFS’s data replication factor will drop from 3 to 2.2 times thanks to a Distributed Raid File System. DRFS increases protection against corruption and hence reduces the amount of replication necessary to ensure availability, for significant disk, rack, and power saving when dealing with Big Data.  CDH4 will also include Apache Mahout for machine learning and improved versions of Flume, Sqoop, and Hue. There will also be new HBase diagnostic and repair tools and Kerberos for security.

Lipcon also mentioned the direction of Hadoop R&D at Cloudera. Goals for future distributions include encryption, disaster recovery, metadata storage and management, resource management, and MapReduce alternatives for problems outside the framework.