Advanced Testing with Great Expectations

Data Quality is still a persistent issue in today’s modern data pipelines.  The demand for data has increased as has the number of data points being analyzed.  Organizations have more source systems than ever, and data quality is no longer an afterthought, there is an expectation to build data quality in line with the ETL project.  This presentation will explore out of the box test functionality in dbt, data quality testing using Great Expectations and various other open-source packages that can provide a baseline for data quality testing.   It will also dive into some advanced testing topics such as constraint tests, and project level tests.

Presentation Topics:

Overview of dbt Tests Configuring, Running and Storing test results Testing sources, models Great Expectations Overview  Integration with dbt Configuring multiple tests  Additional open-source packages to simplify development Advanced data quality tests  By data typeBy table type ConstraintsProject level tests Monitoring, Reporting & Notifications