Valuable Test Automation Techniques for Data Engineers (with dbt!)

Integrating tests and automation can be difficult and time consuming in data pipelines. It's not uncommon for developers to write their transformations with the intent to circle back and build in automated tests...but might not reserve the time to do so. As a data engineer or analytics engineer, is there a way to build test automation as a part of the ETL/ELT development process?  Why, yes there is!

In this talk, you will learn a development technique that Improving created to help integrate test automation into the development lifecycle more easily, while also having more well-thought-out pipelines. You'll learn the different types of testing that are required in data engineering, a process for defining the necessary tests, while also building in an iterative/agile manner. 

This talk will use dbt as the data transformation engine, but the techniques have been successfully applied using SQL, python, pyspark, and other languages at Improving's clients.