Ankit V.

LakeFlow + Delta Live Tables + Serverless Autoscaling Compute sound like a perfect solution to load all required data with data expectations / monitoring / alerting to the Lakehouse :)

12

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

If you are an avid reader of Snowflake release notes, you may have noticed that we have quietly released a number of (in my opinion) pretty cool #sqldelight features in the past few weeks.  If you haven't yet had a chance to read the release notes (shame on you!) here are some quick highlights. We launched #sql #lambda capabilities, with support for FILTER and TRANSFORM of semi-structured data types.  With these higher-order functions, developers now have a more concise and readable way of writing SQL without using FLATTEN, and can avoid the need to use UDFs for this type of logic. https://lnkd.in/gzEP2P_d We’ve added support for ANY as a keyword for the PIVOT function. You can now create a pivot table dynamically, without needing to specify the pivot values explicitly. If you liked GROUP BY ALL, you will love PIVOT ANY! https://lnkd.in/grhqaKBY There are three new columns in the QUERY_HISTORY view: QUERY_RETRY_TIME, QUERY_RETRY_CAUSE, and FAULT_HANDLING_TIME. These new columns help administrators identify workload and/or warehouse changes (e.g. changing size, decreasing concurrency) on warehouses that exhibit a large number of retries. https://lnkd.in/gVEqBBDd If you like these features (or want to see others) let us know! cc: Tim Sander Qinyi Ding #datacloud #datawarehousing #dataengineering #datascience #analytics  

279

11 Comments

 I’m excited to speak along with Saptarshi Mukherjee at @Snowflake’s #DataCloudSummit ❄️in June. Stop by my session, Unify Customer Data, Deliver Exceptional Experiences and Delight Customers with Snowflake and Salesforce, at 03-June 3:30 PM to learn more about Salesforce Bi-Directional Zero Copy Integration and using DOCAI to analyze unstructured files to achieve C360 https://lnkd.in/gC8kHmVr

35

1 Comment

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Comments

A recent Snowflake Automatic Performance enhancement in our core execution engine has led to improved query plan quality for specific query patterns - this has resulted in an 8x speedup, on average, for relevant workloads. This feature is on by default and customers benefit from it automatically, without needing to manually adjust any configurations. Specifically, this new capability - Hierarchical Selectivity Estimates - skips computing the selectivity of predicates based on the table-level aggregated statistics, and instead drills down and applies our estimation logic on the micropartition-level statistics. Compared to table-level statistics, our approach provides high quality table statistics while allowing for efficient updates to the data. As a result of this improvement, we see query duration improved by an average of 8x for relevant workloads, with the largest improvement being over 100x. Not bad! Want to learn more? Check out this blog from Adem Khachnaoui, Rudi Leibbrandt, and Di Wu. PS: This is just one of many such features on our continuous quest to accelerate performance, with a particular focus on turbo-charging the core database engine. The net result: faster queries and lower costs for our customers. https://lnkd.in/gBXV6-2d 

296

6 Comments

A common misconception with AI and machine learning: “all of the resources, infrastructure and technology" it takes to get started. With Databricks ML Flow, that’s simply not the case. In the video ⬇️ Kenny Shaevel breaks down how Databricks makes it easier to manage and track your models while ensuring full transparency, traceability, and governance—without the headache. Kenny covers everything from: - Simplifying Machine Learning Implementation - Utilizing AutoML for Quick Data Evaluation - The Importance of Feature Engineering - Leveraging Feature Store for Reusability - Governance and Traceability with the Model Registry - Seamless Integration and Application of Models Excited for more? Kenny will be diving even deeper into Databricks in our upcoming live stream series. Keep an eye out for details coming soon! 👀 #machinelearning #databricks #dataanalytics #MLFlow #AutoML #featurestore #dataengineering #techinnovation

5

For those that are setting up Databricks with Unity catalog on Amazon Web Services (AWS) , this is a fantastic video which covers step by step to setup AWS roles, policy and identity management to Databricks. https://lnkd.in/g2pwymdj

4

𝗦𝗲𝗺𝗶-𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗱𝗮𝘁𝗮 𝗱𝗼𝗲𝘀𝗻'𝘁 𝗵𝗮𝘃𝗲 𝘁𝗼 𝗯𝗲 𝘁𝗼𝗼 𝗵𝗮𝗿𝗱   Working with varied data formats that have complex or unclear structures is often challenging. Today's data ecosystem deals with more than just traditional rows and columns, with modern databases needing to process complex formats like JSON, XML, and Parquet.   Semi-structured data provides the flexibility to work with dynamic data without requiring a rigid schema. JSON has become popular because it's easy to read, lightweight, and works well for sharing data between web apps, APIs, and databases.   Snowflake has features that make working with JSON easier and faster, with less hassle and more insights.    To help with this, here's a 7,000-word guide on how to query JSON data in Snowflake. ⤵️   Comment what you'd like to learn about MongoDB and JSON data next, and we'll publish it. 😁 #dataingestion #blog #guide #snowflake #query #json #data #datazip #database

14

It was an amazing experience to collaborate with Animesh Kumar to write up this article, from whom I have learnt a lot in the past 3 years! Please read it through and let us know what you think. Follow Modern Data 101 for more such articles. #datastrategy #dataproducts #dataplatform

13

1 Comment

🔥 just in time for the long weekend... 🔥 Databricks Machine Learning models NOW IN EXCEL ! Custom functions call Databricks models directly in Excel. No code. No Databricks knowledge. Our UI guides users to populate the required data inputs. Now anyone can harness Machine Learning models -- investment portfolios; predict outcomes; patient records and experimental data sets. Update entire spreadsheets with Databricks results. https://lnkd.in/e3wcFbG7

2

The Ravit Show covered Snowflake Summit as Press this week and it was incredible! Sridhar Ramaswamy, CEO, Snowflake is such a visionary and it was such a pleasure to meet him in person! Here are some game-changing announcements by Snowflake this week -- 1. Polaris: - Meet Polaris, Snowflake’s latest platform for handling massive data. Think faster, bigger, and easier to use. Polaris is built on Apache Iceberg, making it a vendor-neutral, open catalog that promises seamless data management across different tools and platforms. 2. Snowflake Native App Framework: - Developers, rejoice! You can now build and deploy apps right within Snowflake. This means quicker development and more powerful data apps at your fingertips. It’s a big step towards making the Snowflake Data Cloud a comprehensive platform for all your data needs. 3. Unified Cross-Cloud Governance: - Managing data across different clouds just got a whole lot simpler. Snowflake’s new governance framework keeps your data policies consistent no matter where your data lives. This ensures compliance and security without the usual headaches. 4. Snowpark Container Services: - This is a big one. Now you can run containerized applications directly in Snowflake. It supports all your favorite languages and tools, making your data workflows super flexible. Imagine the possibilities with enhanced data processing and machine learning capabilities right within your data cloud. 5. Expanded Python Capabilities: - For all the Python enthusiasts out there, Snowflake is doubling down. More integrations with popular libraries and frameworks mean you can do even more within the platform. This makes data science and analysis smoother and more efficient. 6. New Marketplace Offerings: - The Snowflake Marketplace is growing! New data sets and third-party apps are now available to supercharge your analytics and insights. Whether you need specific industry data or advanced analytical tools, the Marketplace has got you covered. 7. Performance Enhancements: - Speed and efficiency are the names of the game. Snowflake’s latest updates mean faster queries and more efficient data processing to keep up with modern data needs. These enhancements ensure that your data operations run smoothly, even as your data volumes grow. 8. Partnerships and Integrations: - Snowflake is teaming up with top tech companies like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) to bring even more power to the Data Cloud. New integrations announced today are set to expand what you can do with Snowflake. These partnerships will drive innovation and provide users with a richer set of tools and capabilities. These updates show just how committed Snowflake is to pushing the boundaries in data management and analytics. #data #ai #snowflakesummit #snowflakeflake2024 #theravitshow

170

14 Comments

I love coming into new areas and identifying how we can improve the #data ecosystem by implementing Data Product strategies! A few ways to achieve success in your Data Product journey: 1. Don't just look external, look internal as well. Data products revolutionize operational efficiency by simplifying system to system contracts. 2. Automate data workflows and adopt real-time processing. This helps eliminate manual tasks, reducing costs and freeing up valuable resources. 3. Streamline your consumption patterns. This unlocks faster decision-making in operational and analytical uses. Embracing data products means you enhance agility, optimize processes, and ensure your business runs like a well-oiled machine from end-to-end. Anyone else happen to love fine tuning data ecosystems as much as I do? #Follow along to learn more about how Data Products can unlock your potential! #productmanagement #dataproducts #data

22

2 Comments

Senior Consultant and machine learning expert, Kenny Shaevel, breaks down 3 ways Databricks is helping clients implement ML best practices through MLflow. - Practical use case identification: AutoML helps to quickly evaluate the data you already have available to see how well it would work for a machine learning app. - Repeatability of complex models: Feature Store allows you to “register” features you’ve engineered to use in multiple models as needed. - Governance: Model Registry provides governance and traceability to ensure everyone knows how the models were trained and meant to be used Watch Kenny’s 2-min video for the details! #Databricks #MachineLearning #MLFlow #DataScience #AutoML #FeatureEngineering #ModelRegistry #ExpertInsights

27

1 Comment