Rakhi Jain

If you are an avid reader of Snowflake release notes, you may have noticed that we have quietly released a number of (in my opinion) pretty cool #sqldelight features in the past few weeks.  If you haven't yet had a chance to read the release notes (shame on you!) here are some quick highlights. We launched #sql #lambda capabilities, with support for FILTER and TRANSFORM of semi-structured data types.  With these higher-order functions, developers now have a more concise and readable way of writing SQL without using FLATTEN, and can avoid the need to use UDFs for this type of logic. https://lnkd.in/gzEP2P_d We’ve added support for ANY as a keyword for the PIVOT function. You can now create a pivot table dynamically, without needing to specify the pivot values explicitly. If you liked GROUP BY ALL, you will love PIVOT ANY! https://lnkd.in/grhqaKBY There are three new columns in the QUERY_HISTORY view: QUERY_RETRY_TIME, QUERY_RETRY_CAUSE, and FAULT_HANDLING_TIME. These new columns help administrators identify workload and/or warehouse changes (e.g. changing size, decreasing concurrency) on warehouses that exhibit a large number of retries. https://lnkd.in/gVEqBBDd If you like these features (or want to see others) let us know! cc: Tim Sander Qinyi Ding #datacloud #datawarehousing #dataengineering #datascience #analytics  

279

11 Comments

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

Reproducibility, debugging and compliance across the full lifecycle of ML assets just got so much easier. ML Lineage for end-to-end workflows is now in public preview in Snowflake! 👏 Developers can trace lineage of features, datasets, and models from prototype to production all from a built-in UI. Learn more: https://okt.to/1Gj6W7

95

2 Comments

🔥 just in time for the long weekend... 🔥 Databricks Machine Learning models NOW IN EXCEL ! Custom functions call Databricks models directly in Excel. No code. No Databricks knowledge. Our UI guides users to populate the required data inputs. Now anyone can harness Machine Learning models -- investment portfolios; predict outcomes; patient records and experimental data sets. Update entire spreadsheets with Databricks results. https://lnkd.in/e3wcFbG7

2

Tomorrow, join Databricks for a hands-on session exploring "Data Intelligence with Databricks on AWS." Discover the best practices for implementing comprehensive data analytics, data engineering, and data science lifecycle on the lakehouse architecture with Databricks on Amazon Web Services (AWS). During this live, virtual hands-on lab you’ll learn how to: ·        Access all your data — structured, semi-structured, unstructured — with a Data Intelligence Platform. ·        Use Databricks #SQL to query and visualize data in your #lakehouse architecture. ·        Train models and create predictions with Databricks on AWS. ·        Track experiments and tune hyperparameters with #MLflow. ·        Deploy and serve models with MLflow and other AWS services. https://lnkd.in/eKvTaekz

5

I'm #hiring for a couple of open roles in the Data Science/Machine Learning field (contract). Candidates must have strong experience in 1) Building robust ETL scripts using SQL & Python to handle diverse structured and unstructured data sources. 2) Creating and maintaining real-time streaming data platforms using technologies like Apache Kafka, Spark, or Flink. 3) Designing data engineering solutions, data warehouses, and advanced analytics using SQL & Python. Familiarity with ML/AI modeling is a plus. #machinelearning #datascience #statisticalmodeling #python #sql #realtimeanalytics #ml #mlops #aiml

47

12 Comments

A common misconception with AI and machine learning: “all of the resources, infrastructure and technology" it takes to get started. With Databricks ML Flow, that’s simply not the case. In the video ⬇️ Kenny Shaevel breaks down how Databricks makes it easier to manage and track your models while ensuring full transparency, traceability, and governance—without the headache. Kenny covers everything from: - Simplifying Machine Learning Implementation - Utilizing AutoML for Quick Data Evaluation - The Importance of Feature Engineering - Leveraging Feature Store for Reusability - Governance and Traceability with the Model Registry - Seamless Integration and Application of Models Excited for more? Kenny will be diving even deeper into Databricks in our upcoming live stream series. Keep an eye out for details coming soon! 👀 #machinelearning #databricks #dataanalytics #MLFlow #AutoML #featurestore #dataengineering #techinnovation

5

I've successfully obtained the Databricks Certified Data Engineer Associate certification. This certification validates proficiency in: → Databricks Lakehouse Platform → Multi-hop ETL with Apache Spark SQL and Python → Batch and incremental data processing → ETL pipelines and Databricks SQL queries production I'm excited to apply these skills to drive more value in data engineering/data science projects at CTI DATA and beyond. I'm now 2X Databricks. Never stop learning! Let's go! #DatabricksLearning #DataEngineering #Databricks #ProfessionalDevelopment

24

1 Comment

🚀 Looking for new opportunities in Data Engineering 🚀 I’m a highly skilled GCP Data Engineer with 3+ years of experience, specializing in ETL/ELT pipeline development, BigQuery analytics, and DevOps automation. My expertise spans across the Telecom and Payroll domains, where I’ve successfully built batch and streaming data pipelines on Google Cloud and integrated Looker dashboards for real-time insights. Proficient in using tools like Airflow, Terraform, Docker, and more, I’ve optimized workflows, reduced cloud costs, and ensured data quality. 🔑 Key skills: GCP, Python, SQL, Cloud Run, Dataflow, Terraform, CI/CD 🏆 Certification: GCP Associate Cloud Engineer 💼 Recent Projects: 1. Designed a comprehensive payroll reconciliation system, leveraging BigQuery and Looker for real-time analytics. 2. Developed ETL pipelines for one of the largest European telecom companies, managing large datasets for performance optimization and reporting. I’m now looking to bring my skills to the next level with Google Data Engineering roles! Let's connect if you have any opportunities! #DataEngineering #GCP #Looker #ETL #BigQuery

6

Senior Consultant and machine learning expert, Kenny Shaevel, breaks down 3 ways Databricks is helping clients implement ML best practices through MLflow. - Practical use case identification: AutoML helps to quickly evaluate the data you already have available to see how well it would work for a machine learning app. - Repeatability of complex models: Feature Store allows you to “register” features you’ve engineered to use in multiple models as needed. - Governance: Model Registry provides governance and traceability to ensure everyone knows how the models were trained and meant to be used Watch Kenny’s 2-min video for the details! #Databricks #MachineLearning #MLFlow #DataScience #AutoML #FeatureEngineering #ModelRegistry #ExpertInsights

27

1 Comment

🚀 Exciting Highlights from Snowflake BUILD Bangalore Dec 19th 2024! 🚀 Attending the Snowflake BUILD event on December 19th was an amazing experience, packed with insights into the latest advancements and powerful features of Snowflake. Here are some of the top new features and key learnings from the event: 1. Apache Polaris - Catalog Revolutionizing how we manage and access data with an open table format and REST Catalog for seamless integration. 2. Storage Enhancements Optionally archiving tables to optimize storage costs and manage data retention effectively. 3. Document AI - Copy INTO Simplifying data ingestion with new capabilities for handling document-based data. 4. Snowflake Horizon Catalog Enabling unified governance processes, providing full visibility across your data ecosystem. 5. Snowflake Notebooks A powerful tool for collaborative data science and analytics directly within Snowflake. 6. Observability: Generative AI & ML New capabilities for deep insights, leveraging AI and ML for enhanced observability. 7. SPCS - Snowpark Container Services Enabling more flexible and efficient data processing with containerized workloads. 8. Internal Marketplace - GET Button Simplifying access to shared data and services with a unified internal marketplace. 9. Egress Cost Optimizer Optimizing costs associated with data egress to improve operational efficiency. 10. Share Fine-Tuned LLMs Empowering users to share their fine-tuned large language models (LLMs) for advanced analytics. 11. Monetize AI Apps Enabling the creation and monetization of AI-powered applications for business growth. 12. Hybrid Tables & Unistore Solving concurrency challenges and enhancing performance with hybrid storage and compute solutions. 13. MODIN - Scalable Pandas Scalable, distributed computing for Pandas users, now leveraging RAY, DASK and SNOWPARK environments. 14. Altair Library for Visualization Bringing advanced visualization capabilities to Snowflake for data-driven insights. 15. Snowflake DevOps Capabilities Enhanced DevOps features with capabilities like CREATE OR ALTER, EXECUTE IMMEDIATE, Snowflake CLI, GIT Integration, Python APIs, and more. These innovations are taking data management, governance, and analytics to the next level. The future of data is truly exciting with Snowflake's cutting-edge solutions! 🌐 #Snowflake #SnowflakeBUILD #DataInnovation #AI #MachineLearning #DataGovernance #DAP #Ericsson #DAPPLATFORM

48

1 Comment

I like Databricks' dynamic resource allocation; I think it is great. However, I would be cautious when a developer sets the maximum worker nodes to 20. You risk wasting resources and spending a lot of money because the allocation and deallocation of resources are not simple processes behind the scenes. Often, it takes a while to deallocate the newly added executors, leading to overpayment for low utilization. A quick look at the Metrics UI can provide insight into this issue. Instead, try to resize your cluster compute to a larger size, ensuring you do not exceed a maximum of 8 worker nodes in your upper limit setting. #databricks #dataengineer

1

Migrating a legacy data warehouse workload to a cloud analytics platform is no easy task. It requires a robust implementation plan, strong change management skills, and massive amounts of code conversion and data movement. This is why Snowflake offers #SnowConvert to our customers and partners - SnowConvert is a high-fidelity conversion tool that creates a complete semantic model based on your codebase and automates huge portions of the DW migration process. Today, we are pleased to announce an expansion of SnowConvert’s capabilities. You can now convert tables and views for #Oracle, #Teradata and #SQLServer with a self-service product experience. This enhancement enables you to convert your entire data model and accelerate your transition to Snowflake faster than ever before. Want to learn more? Check out this blog post from Federico Zoufaly for more detailed information on the inner workings of SnowConvert, so you can understand its real power and value at the core of a migration project from a legacy data warehouse platform. https://lnkd.in/g9XwQkzR (Link to get access is in the comments)

231

3 Comments

LakeFlow + Delta Live Tables + Serverless Autoscaling Compute sound like a perfect solution to load all required data with data expectations / monitoring / alerting to the Lakehouse :)

12

A recent Snowflake Automatic Performance enhancement in our core execution engine has led to improved query plan quality for specific query patterns - this has resulted in an 8x speedup, on average, for relevant workloads. This feature is on by default and customers benefit from it automatically, without needing to manually adjust any configurations. Specifically, this new capability - Hierarchical Selectivity Estimates - skips computing the selectivity of predicates based on the table-level aggregated statistics, and instead drills down and applies our estimation logic on the micropartition-level statistics. Compared to table-level statistics, our approach provides high quality table statistics while allowing for efficient updates to the data. As a result of this improvement, we see query duration improved by an average of 8x for relevant workloads, with the largest improvement being over 100x. Not bad! Want to learn more? Check out this blog from Adem Khachnaoui, Rudi Leibbrandt, and Di Wu. PS: This is just one of many such features on our continuous quest to accelerate performance, with a particular focus on turbo-charging the core database engine. The net result: faster queries and lower costs for our customers. https://lnkd.in/gBXV6-2d 

296

6 Comments

A big thank you to Sumit Mittal and TrendyTech on my Data Engineering learnings. In this module, a lot of ground was covered in understand and working with - - RDDs and Parallelize - Narrow and Wide Transformations - reduceByKey & groupByKey - Broadcast Joins - Repartition and Coalesce - Cache #bigdataengineer #apachespark #sql #aws #azure

54

6 Comments

𝐔𝐧𝐥𝐨𝐜𝐤𝐢𝐧𝐠 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐚𝐧𝐝 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐰𝐢𝐭𝐡 𝐒𝐩𝐚𝐫𝐤 𝐉𝐨𝐛𝐬 𝐨𝐧 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 🚀 As organizations delve deeper into data driven decision making, the efficiency and performance of Spark jobs hosted on platforms like Databricks become crucial factors in driving success. Optimizing Spark jobs ensures not only faster execution but also cost-effectiveness and resource utilization. 💰📊 There are several recommendations as listed below that help to improve the efficiency but they cannot be applied in every scenario. 📍 It's very important to understand when and where to apply these strategies to reap the real benefits and this document has been very helpful to me to build the cost and resource efficient data pipelines on Databricks. I hope you find it useful as well. 👉 Deletion Vectors 👉 Join Strategies 👉 DBR and Photon engine 👉 Partitions 👉 File Sizes 👉 Zorder 👉 Stats and many more..... Please let me know the strategies you use to performance tune etl jobs through comments. Document credits: Databricks #databricks #sql #dataengineer #cloud #datalake #analytics #spark #optimization #efficiency #engineer #bigdata

2

Excited to see the Apache XTable ecosystem and use cases expand when Titans like Microsoft Fabric and Snowflake are able to build interoperability with the project. BIG MOVE for open data! 🔥 Quote from the ❄ Snowflake blog: "Fabric will be able to store data in Iceberg format in OneLake via Apache XTable translation in OneLake" #ApacheXTable #ApacheIceberg #DeltaLake #datalakehouse #Snowflake #MicrosoftFabric #OneLake

36

1 Comment