Austin U, MSc.’s Post

🤔 Are you highly analytical, naturally curious, and always ambitious? 🛠️ Passionate about building systems and infrastructure? 📈 Do you dream about not just more data, but better data? Become the next mover and shaker at Snowflake — check out the new Data Platform team page to see who we are, what we do, and how we impact the data revolution: https://okt.to/aR0zYU #lifeatsnowflake

🤔 Are you highly analytical, naturally curious, and always ambitious? 🛠️ Passionate about building systems and infrastructure? 📈 Do you dream about not just more data, but better data? Become the next mover and shaker at Snowflake — check out the new Data Platform team page to see who we are, what we do, and how we impact the data revolution: https://okt.to/lWo93T #lifeatsnowflake

Red pill or Blue pill? Morpheus opted to perform some clustering in Snowflake. He has presented the below two situations of clustering to choose from to Neo. Which one should Neo go for? Red or Blue? (Note : MP in the snapshot means Micro-partition) How do you decide that? Remember Snowflake's goal? Scan the least number of partitions to return the desired query results. So, what's your answer? Red or Blue? Discover Learning experiences that prioritizes action over learning at #SnowflakeAcademy at Enqurious Academy Start here : https://lnkd.in/gD_ruthp

🌟 Spending my nights and weekends strengthening my Data Engineering skills with Zach Morris Wilson's Bootcamp! 🌟 The first week has been both challenging and rewarding. I had to go through the labs several times to truly master the concepts, but the satisfaction of finally being able to do it all by myself is unmatched! With Thanksgiving here, I’m taking the opportunity to catch up on the bootcamp and dive deeper into some new Data Engineering concepts. Here are some of my key takeaways so far: 💡 Advanced Data Types for Compact Storage -Arrays, Struct, and Map are incredibly useful for making datasets more compact. -Compact datasets not only save storage but also reduce querying time, optimizing data pipelines. ⚖️ Tradeoffs in Design -Designs like Cumulative Tables simplify access to historical data and save space but can make querying more complex for certain use cases. -Finding the right balance between simplicity, compactness, and performance is crucial. 📊 Knowing Your Data Customers Whether it’s Analysts, Engineers, ML Models, or Business Users, tailoring your data to meet their needs ensures the most effective outcomes. Would highly recommend this bootcamp to anyone looking to strengthen their Data Engineering skills. Looking forward to more learnings and challenges ahead! 🚀 #DataEngineering #Learning #DimensionalModeling

When a model underperforms, it can be difficult to understand why. Understanding and explaining model behavior is typically done using Shapley values and plays a key role in ensuring quality of the model before it goes to production, and debugging the model when it underperforms in production. Snowflake ML Explainability is now in Public Preview! Senior Product Manager at Snowflake, Avinash Joshi is here to share: 1️⃣ How to use Shapley values 2️⃣ How to compute Shapley values within Snowflake 3️⃣ The benefits of using Snowflake to compute Shapley values Read the blog to learn more: https://lnkd.in/gjN7SnF4

I'm excited to share that I’m currently enrolled in a Data Engineering Bootcamp offered at dataexpert.io led by the inspiring Zach Morris Wilson! During the SCD lecture, a slide really caught my attention(attached below), especially the point about using start_date > without a corresponding end_date <. In some of my pipelines, I use a start_date to push data greater than the max(date) into a blob. For subsequent runs, I check the max(date) in the blob and push only new data that exceeds this value i.e. the max(date) in the blob. This approach has been reliable so far, and I’m confident it ensures idempotency. However, it made me wonder if explicitly defining an end_date would be a better practice. For instance, I could use utcnow() as the end_date for every run. What are your thoughts? Does my current approach sufficient, or is incorporating an end_date a step worth adding? I’d love to hear your insights! Zach Morris Wilson

🚀 Exciting Announcement! 🚀 📊 Embarking on a Journey: Starting the #100DaysOfDataScience Challenge 📊 I'm thrilled to announce that I've taken the plunge and committed myself to the #100DaysOfDataScience challenge! 🎉 As a passionate advocate for leveraging data to drive insights and innovation. 💡Over the next 100 days, I'll be diving deep into various aspects of data science, from exploring machine learning algorithms to mastering data visualization techniques and everything in between. Each day will bring new challenges and opportunities for discovery, and I can't wait to see how far I'll progress by the end of this journey. 🌟 But I'm not doing this alone! I invite you all to join me on this adventure. 📚 Throughout the challenge, I'll be documenting my progress and sharing insights, resources, and tips along the way. You can follow my journey on GitHub, where I'll be posting daily updates and code snippets: 🔗https://lnkd.in/gSntBA3P Are you ready to join the #100DaysOfDataScience challenge? Let's harness the power of data to drive meaningful change and innovation! 🌐💼 #DataScience #MachineLearning #DataVisualization #GitHub #100daysofdatascience

🚀 𝐂𝐨𝐦𝐩𝐥𝐞𝐭𝐞𝐝 𝐖𝐞𝐞𝐤 𝟔 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐙𝐨𝐨𝐦𝐜𝐚𝐦𝐩! Wrapping up another dynamic week at the Data Engineering Zoomcamp, and I'm thrilled to announce the completion of Week 6! 🌟 This week delved deeper into 𝗦𝗽𝗮𝗿𝗸, harnessing the power of Pyspark and Spark SQL to handle data. Connecting Spark directly to 𝐆𝐨𝐨𝐠𝐥𝐞 𝐂𝐥𝐨𝐮𝐝, to leverage its storage and computation power, giving us the possibility to scale our jobs Here's a glimpse of what we covered: 🧠 𝐒𝐩𝐚𝐫𝐤 𝐈𝐧𝐭𝐞𝐫𝐧𝐚𝐥𝐬: Explored the inner workings of Spark, gaining a deeper understanding of its architecture and mechanisms. 📊 𝐏𝐲𝐬𝐩𝐚𝐫𝐤 & 𝐒𝐩𝐚𝐫𝐤 𝐒𝐐𝐋:: Leveraged Pyspark and Spark SQL to execute data operations. 🌐 𝐂𝐥𝐮𝐬𝐭𝐞𝐫 𝐂𝐫𝐞𝐚𝐭𝐢𝐨𝐧: From local clusters to setting up a GCP Dataproc cluster, we navigated the process of establishing robust environments for our Spark endeavors. 🔄 𝐁𝐚𝐭𝐜𝐡 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠: Dived into the world of batch processing with Spark, understanding its nuances and applications in real-world scenarios. Thank you Alexey Grigorev for this week, and for the valuable insights. 🙌 Looking forward to diving into the exciting world of streaming processing with Kafka next week! GitHub Repo Link: https://lnkd.in/eYUdEVZ6

I just finished the "Machine Learning with Scikit-Learn" course on Infosys Springboard, and it was a great learning experience! The course helped me understand how machines learn from data. I learned about important concepts like regression, classification, and clustering, and how to use Scikit-Learn to apply these techniques in real projects. The hands-on exercises were very helpful and gave me confidence in working with machine learning models. I’m excited to use these new skills in future projects!

Snowflake Northstar: Don't miss these new free courses on Snowflake on Coursera! Learn how to build applications, data pipelines, AI models and more with Snowflake. Get started with the Intro to Snowflake for Devs, Data Scientists, Data Engineers course, or dive directly into our Introduction to Modern Data Engineering course if you are already familiar with Snowflake. Created by Snowflake’s own developer advocates, the courses are intended to get you started quickly with a heavy emphasis on hands-on exercises and demos. The course materials are free if you select the “Audit Only” option when enrolling. https://lnkd.in/ey7h-hFS #SnowflakeSquad