From the course: End-to-End Data Engineering Project
Join today to access over 24,100 courses taught by industry experts.
Solution: Add a freshness check
From the course: End-to-End Data Engineering Project
Solution: Add a freshness check
(upbeat music) - [Instructor] To solve this challenge, you should have added a freshness property in your src_big_star_db.yml file similar to this. This configuration ensures that if the data in neither table is older than 24 hours, the freshness check will show a warning. I am using the _airbyte_normalized_at times time column to verify freshness. After adding the freshness key, you can execute the dbt source freshness command. As you can see, I am getting warnings here. The last time I ran the Airbyte sync was more than two days ago. Hence, the check is working as expected. This feature is quite relevant for a production pipeline. The source data is loaded by an external system, Airbyte, but with this check, you can have visibility on the extract and load processes from dbt. If there was a problem with the extract and load and the data was not flowing, you could investigate the issue and avoid working with outdated…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.