Make Apache Hive queries testable inside Docker container.

In my current work we are often using Apache Hive for making the reports for one of the analytics platform. It’s becoming harder and harder to test your queries outside of the clusters and it’s pretty expensive to setup staging environment using Amazon EMR as an example with predefined staging data there. I’ve started to think about to use advantages of containers.

Lets start from the beginning. The goal:

  1. Use the specific version of Apache Hive to have the same environment like cluster does.
  2. Use the test framework on local machine for easily define fixtures and describe the test cases.
  3. Use hive shell to access the current tests and review log files in case of the issues.

All these goals could be done by using Docker, RSpec, Makefile.

Lets define Makefile, it should be our entry to run test cases:

Now you should run bundle install and then make diectoryspec inside of your project. We are ready to define structure for RSpec test cases and base classes to make test runnable inside of containers.

Usually I am starting from making the prototype of the _spec.rb with test cases.

I defined execute_query(query, options = {}), execute(query_path, options = {})and fetch_query(query, options = {})methods. load_data creates query to INSERT OVERWRITE from txt file to your table required by test cases. I’ve done it by creating text format table duplicate for the prod version of the table. Then it will load data directly to table without partition in text format and then copies data to partitioned(or not) any format prod version table.

Now I will define spec_helper.rb with examples how to run the queries.

Basic workflow for running tests:

That’s it. But it’s really helpful in our cases. You don’t need to have staging clusters. It’s possible to define different hive versions in separate containers and migrate your queries adapting to new versions.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store