Le Thi Phuong THAO’s Post

💡 Snowflake ETL Tip: Optimize Data Transfer with Snowpipe Auto-Ingestion! 🌊 Enhance the efficiency of your ETL pipeline by utilizing Snowpipe for automatic data ingestion. Snowpipe enables real-time loading of data into Snowflake from cloud storage platforms like S3, Azure Blob Storage, or Google Cloud Storage, eliminating the need for manual data loading processes and ensuring near real-time data availability. Here's an example code snippet showcasing how to set up Snowpipe for auto-ingestion: -- Create a Snowpipe for auto-ingest from a stage CREATE PIPE my_snowpipe AUTO_INGEST = TRUE AS COPY INTO my_table FROM @my_stage; -- Monitor Snowpipe ingestion status SHOW PIPE my_snowpipe; #Snowflake #ETL #Snowpipe #AutoIngestion #RealTimeData #DataIntegration #TechTips #CloudStorage #DataLoading #LORSIVTechnologies Harness the power of Snowpipe for seamless and automated data ingestion, enabling near real-time updates in your Snowflake data warehouse. Share your experiences with Snowpipe or tag a colleague interested in optimizing their data transfer processes! Don't forget to follow #LORSIVTechnologies for more Snowflake and ETL tips and insights!

💡 Snowflake ETL Tip: Optimize Data Transfer with Snowpipe Auto-Ingestion! 🌊 Enhance the efficiency of your ETL pipeline by utilizing Snowpipe for automatic data ingestion. Snowpipe enables real-time loading of data into Snowflake from cloud storage platforms like S3, Azure Blob Storage, or Google Cloud Storage, eliminating the need for manual data loading processes and ensuring near real-time data availability. Here's an example code snippet showcasing how to set up Snowpipe for auto-ingestion: -- Create a Snowpipe for auto-ingest from a stage CREATE PIPE my_snowpipe AUTO_INGEST = TRUE AS COPY INTO my_table FROM @my_stage; -- Monitor Snowpipe ingestion status SHOW PIPE my_snowpipe; #Snowflake #ETL #Snowpipe #AutoIngestion #RealTimeData #DataIntegration #TechTips #CloudStorage #DataLoading #LORSIVTechnologies Harness the power of Snowpipe for seamless and automated data ingestion, enabling near real-time updates in your Snowflake data warehouse. Share your experiences with Snowpipe or tag a colleague interested in optimizing their data transfer processes! Don't forget to follow #LORSIVTechnologies for more Snowflake and ETL tips and insights!

Proud to share that I've completed "Data Warehouse - The Ultimate Guide" by Nikolai Schuler! Learned everything from data modeling and ETL processes to advanced topics like OLAP cubes and cloud data warehouses. Ready to leverage this knowledge in real-world projects! #DataWarehouse #ETL

How to build Data Pipelines: Old way ▶ Use a relational database ▶ Store data on-premise ▶ Create individual packages for each table ▶ Design everything up front, build after planning ▶ Extract > Transform > Load ▶ Consider data analytics only New Way ▶ Use a modern data platform (e.g. Fabric, Databricks, Snowflake) ▶ Store data in the cloud ▶ Create data pipeline processes per type of data (reuse baby!) ▶ Plan & Develop in sprints ▶ Extract > Load > Transform (Medallion architecture) ▶ Allow for data science and analytics use cases Modern (New) > Legacy (Old) 💚 #datapipeline #medallion #datalake #datawarehouse #modern #bi #dataengineer

How to build Data Pipelines: Old way ▶ Use a relational database ▶ Store data on-premise ▶ Create individual packages for each table ▶ Design everything up front, build after planning ▶ Extract > Transform > Load ▶ Consider data analytics only New Way ▶ Use a modern data platform (e.g. Fabric, Databricks, Snowflake) ▶ Store data in the cloud ▶ Create data pipeline processes per type of data (reuse baby!) ▶ Plan & Develop in sprints ▶ Extract > Load > Transform (Medallion architecture) ▶ Allow for data science and analytics use cases Modern (New) > Legacy (Old) 💚 #datapipeline #medallion #datalake #datawarehouse #modern #bi #dataengineer

5 Traditional Data Warehouses Challenges & Solutions with Snowflake Data Cloud 💸 High storage and ETL processing costs? Offload ETL to Snowflake for cost reduction and increased data warehouse capacity. ❄️ Struggling with unused data driving up costs? Move cold data to Snowflake for efficient storage and EDW usage. 🔄 Need help integrating new data sources? Snowflake supports diverse data types for enhanced analytics. 🛠️ Dealing with data quality issues? Implement data quality checks with Snowflake for reliable data and analytics. 🏛️ Facing data governance challenges? Establish accountability and decision-making with Snowflake's governance features. #datawarehouse #snowflake #snowflakedatacloud #datagovernance #dataquality #ETL #

Thrilled to announce the completion of my Data Warehousing Concepts certification from DataCamp! This certification offered a robust foundation in modern data warehousing, equipping me with essential knowledge in several key areas: - Data Warehouse Essentials: Developed a strong understanding of data warehousing fundamentals, distinguishing data warehouses from data marts and data lakes, and gaining insight into the pivotal roles within data warehousing projects. - Architectural Approaches: Analyzed typical warehouse architectures, comparing Bill Inmon’s top-down approach with Ralph Kimball’s bottom-up methodology, and explored the unique functionalities of OLAP versus OLTP systems. - Data Modeling Techniques: Acquired skills in data structuring through star and snowflake schemas, understanding fact and dimension tables, and managing slowly changing dimensions using Kimball's methodology. - Implementation & Data Prep: Examined ETL and ELT processes, and evaluated best practices for implementing data warehouses across on-premise, cloud, and hybrid environments. Thank you, DataCamp, for a valuable and in-depth learning experience! #DataWarehouse #Analytics #DataArchitecture #BusinessIntelligence #Cloud #DataCamp #DataTransformation

The secret’s out: Capella Columnar is GA! 🏛️ Capella Columnar enables real-time, zero ETL JSON-native data analytics that run independently from operational workloads—all within one database platform. Tune in Sept. 17 & 18 as we debut it's features and answer your questions#couchbase #cloud #database #NoSQL #JSON #DBaaS#AI #vectorsearch #edgeAI

What’s the best way to move #Kafka data to #Snowflake? 🤔 Find out in this whitepaper, where we discusses the reasons for moving data from Kafka to Snowflake, such as the need to make real-time data available for SQL queries and self-service visualization. We will also review and demonstrate important considerations when building a data pipeline from Kafka to Snowflake, including: ✅ Data consistency ✅ Schema evolution ✅ Observability ✅ Architecture consideration Furthermore, we will take a comprehensive overview of the options involved in moving your data, helping readers make informed decisions based on their specific requirements and use cases. This guide is suitable for: ☑️ Data engineering leaders who want to understand how to solve practical big data ingestion challenges. ☑️ CTOs who want to discover cloud architecture best practices. ☑️ Data practitioners who want to enrich their knowledge about big data ingestion. Get your copy here 👉 https://lnkd.in/eFXitUZA #DataEngineering #DataPipelines #CloudArchitecture

Accelerate your journey to Snowflake with Next Pathway's detailed data migration checklist. Our step-by-step guide helps you efficiently transition your legacy EDW, Data Lake, and ETL processes to Snowflake's powerful cloud platform. Our expertly crafted checklist covers crucial steps, from initial data categorization to final migration verification, ensuring a seamless and successful transition. Key aspects include: • Data inventory and classification • Resource planning and allocation • Migration approach selection • Comprehensive data validation • Optimization of Snowflake's unique features • Smooth cut-over strategy implementation By leveraging our industry-leading expertise, you can minimize migration risks, enhance data quality, and significantly accelerate your migration process to Snowflake. Download our Snowflake Data Migration Checklist now and master your cut-over strategy: https://bit.ly/3IL4lD9 #NextPathway #SHIFTCloud #CloudAutomation #CloudMigration #Snowflake #Onelake #DataFactory #AzureSynapes #Informatica #ETL #AI #Aiautomation #GenerativeAI