Working in a hybrid infrastructure brings unique challenges! I am thrilled to share a recent milestone in optimizing our Business Intelligence data infrastructure! We successfully enabled multi-database support in our Airflow and dbt Labs environment, leveraging PostgreSQL for OLTP operations and BigQuery for OLAP workloads. This enhancement allows us to: - Streamline real-time transactional processing in PostgreSQL. - Power advanced analytics and reporting in BigQuery. - Facilitate distinct workflows while maintaining a unified orchestration layer in Airflow. What makes this achievement unique is the challenge of setting up Airflow, dbt, and PostgreSQL on an on-premise server where we have to make sure the scalability, security, networking are well configured. Overcoming these hurdles has resulted in a robust hybrid architecture, empowering us to deliver data-driven insights with efficiency and precision. This is yet another step forward for Lamudi BI! Kudos to all my team members who made this happen: M Daniel Stevanus Muhamad Alfin Eka Rohadi Anisa Nabila Hafizh As'ad Baihaqi Yoza Pratama Stay tuned for our next update—our journey continues as we explore and implement new data infrastructures! #DataEngineering #BigQuery #PostgreSQL #ApacheAirflow #dbt #OnPremise #BI #DataInnovation #HybridCloud
OBI obi
3w
Great insights, Mas Adnan Zahry! Let's catch up soon to discuss how Confluent can help make data your most powerful asset. 💪
Helping SMEs grow 10x with Data and AI 🚀
2wWhat made you take the difficult decision to have Airflow, dbt, and PostgreSQL on an on-premise server?