Templating ETLs is challenging! The creation and maintenance of data pipelines in production require hard work to manage bugs in code and bad data.

I like to propose a data pipeline pattern that can simplify building pipelines while optimizing for data integrity and observability. The workflow is built using open source tools like Singer, Great Expectations, and DBT.

Goals:

• Make ELT simple and fast to implement
• Validate your assumptions of the data before you make it available for use
• Allow analysts/data scientists add pain-free contributions to ELT using SQL
• Generate data documentation, failure logs for quick recovery, and fixes outages in your pipeline

Target Audience:

• Approachable to any level of developer
• Novice data personals interested in starting ELT workflow and learning about different tools of the ecosystem
• Intermediate+ developers interested in supercharging their pipeline with Write Audit Publish pattern and reducing pipeline debt