Apache Drill vs. Apache Spark: What’s The Right Tool for the Job?

March 1, 2016
41 Views

Image

If you’re looking to implement a big data project, you’re probably deciding whether to go with Apache Spark SQL or Apache Drill. This article can help you decide which query tool you should use for the kinds of projects you’re working on.

Image

If you’re looking to implement a big data project, you’re probably deciding whether to go with Apache Spark SQL or Apache Drill. This article can help you decide which query tool you should use for the kinds of projects you’re working on.

Spark SQL

Spark SQL is simply a module that lets you work with structured data using Apache Spark. It allows you to mix SQL within your existing Spark projects. Not only do you get access to a familiar SQL query language, you also get access to powerful tools such as Spark Streaming and the MLlib machine learning library.

Spark uses a special data structure called a DataFrame that represents data as named columns, similar to relational tables. You can query the data from Scala, Python, Java, and R. This enables you to perform powerful analysis of your data rather than just retrieving it. But it’s even more powerful when extracting data for use with the machine learning library. With MLlib, you can perform sophisticated analyses, detect credit card fraud, and process data coming from servers.

As with Drill, Spark SQL is compatible with a number of data formats, including some of the same ones that Drill supports: Parquet, JSON, and Hive. Spark SQL can handle multiple data sources similar to the way Drill can, but you can funnel the data into your machine learning systems mentioned earlier. This gives you a lot of power to analyze multiple data points, especially when combined with Spark Streaming. Spark SQL serves as a way to glue together different data sources and libraries into a powerful application.

Apache Drill

Apache Drill is a powerful database engine that also lets you use SQL for queries. You can use a number of data formats, including Parquet, MongoDB, MapR-DB, HDFS, MapR-FS, Amazon S3, Azure Blob Storage, Google Cloud Storage, Swift, NAS, and more.

You can use data from multiple data sources and join them without having to pull the data out, making Drill especially useful for business intelligence.

The ability to view multiple types of data, some of which have both strict and loose schema, as well as being able to allow for complex data models, might seem like a drag on performance. However, Drill uses schema discovery and a hierarchical columnar data model to treat data like a set of tables, independently of how the data is actually modeled. 

Almost all existing BI tools, including Tableau, Qlik, MicroStrategy, Spotfire, SAS, and even Excel, can use Drill’s JDBC and ODBC drivers to connect to it. This makes Drill very useful for people already using BI and SQL databases to move up to big data workloads using tools they’re already familiar with.

Drill’s JDBC driver lets BI tools access Drill. JDBC lets developers query large datasets using Java. This has a similar advantage that using ANSI SQL does: lots of developers are already familiar with Java and can transfer their skills to Drill.

Easy Data Access in Drill

One of Drill’s biggest strengths is its ability to secure databases at the file level using views and impersonation.

Views within Drill are the same as those within relational databases. They allow a simplified query to hide the complexities of the underlying tables. Impersonation allows a user to access data as another user. This enables fine-grained access to the raw data when other members of your team should not be able to view sensitive or secure data.

Views and impersonation are beyond the scope of Apache Spark.

Conclusion

So which query engine should you choose? As always, it depends. If you’re mainly looking to query data quickly, even across multiple data sources, then you should look into Drill. If you want to go beyond querying data and work with data in more algorithmic ways, then Spark SQL might be for you. You can always test both out by playing around in your own Sandbox environment, which lets you play around with these powerful systems on your own machine.

You may be interested

The State of US Cyber Security
IT
0 shares76 views
IT
0 shares76 views

The State of US Cyber Security

bcornell - May 25, 2017

During the first week of May 2017 President Donald Trump signed a cyber security executive order focusing on upgrading government…

Tips to keep your eCommerce Store Secured against Hackers
IT
0 shares128 views
IT
0 shares128 views

Tips to keep your eCommerce Store Secured against Hackers

Rehan Ijaz - May 25, 2017

“There are risks and costs to a program of action--but they are far less than the long-range cost of comfortable…

The Lessons We can Learn from Bad Data Mistakes Made Throughout History
Big Data
0 shares154 views
Big Data
0 shares154 views

The Lessons We can Learn from Bad Data Mistakes Made Throughout History

Matthew Zajechowski - May 25, 2017

Bad data is costly.  With data driving so many decisions in our lives, the cost of bad data truly impacts…