Ilyes Mehaddi

Dynamic Tables are generally available ! Dynamic tables allow you to create data pipelines that are easy to build, operate, and evolve. With general availability, the following new enhancements are added: Sharing and collaboration: Dynamic tables can now be shared across regions and clouds using Snowflake’s sharing and collaboration features. This makes it easy to share clean, enriched, and transformed data products with consumers in your organization, partner organizations, or the broader data cloud community, ensuring they stay updated according to your specified cadence. Disaster recovery and replication: Dynamic tables now support high availability through Snowflake’s replication infrastructure. You can build your production pipelines with peace of mind knowing that you’re supported with Snowflake’s disaster recovery solutions. Observability: New functionality added for better observability via Snowsight and programmatic interfaces. In Snowsight, there are new account-level views, visibility into warehouse consumption, improved graph and refresh history, and the ability to suspend and resume refreshes. Observability functions now include new account usage views, extended retention of information schema functions and added support for consistent metadata across Snowflake observability interfaces. Data Cloud integrations: Added support for clustering, transient dynamic tables, and governance policies (on sources of dynamic tables and dynamic tables themselves), allowing you to benefit from the best features of the Snowflake Data Cloud. Scalability: You can now create four times more dynamic tables in your account, and ten times more dynamic table sources feeding into another dynamic table. There are no longer any limits on the depth of a directed acyclic graph (DAG) that you can create. Query evolution and SELECT * support: Dynamic tables now automatically evolve to absorb new columns from base tables and incrementally repair without needing to rebuild the dynamic table when new columns are added. New documentation: We’ve added new articles to our documentation covering development best practices, performance optimization guides, troubleshooting pipeline issues, and other improvements. https://lnkd.in/epkbrVTJ Our partner Indexima offers automatic dynamic table creation&maintenance based on query patterns ! Learn more in the interview (in French) https://lnkd.in/eFKDSsxj

33

Here's a secret Snowflake tip 🤫: with the ability to host Notebooks on the new container runtime, we can now install arbitrary packages using !pip. 🔥 Yes, we can also install packages not available on the Anaconda Snowflake repo (with all the risks that come with it 😅). For instance, now you can install the Cinemagoer package and search the IMDb for all the movies containing the word "matrix" in the title 👇

93

3 commentaires

#DataEngineering 𝗣𝗿𝗼 𝗧𝗶𝗽𝘀 𝗳𝗿𝗼𝗺 𝗙𝗮𝗯𝗿𝗶𝗰𝗲 𝗗𝗲𝘀𝗲𝘆𝗻, 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗖𝗵𝗮𝗺𝗽𝗶𝗼𝗻 Join our colleague, Fabrice Deseyn, for a deep dive into the world of Databricks. In a recent interview with Youssef Mrini on the #NextGenLakehouse YouTube channel, Fabrice shares his insights and journey of becoming a Databricks champion. Beyond his champion status, Fabrice provides expert guidance on navigating Unity Catalog migrations. Discover practical tips and strategies to streamline the process and ensure a successful transition. Watch the video now 👉 https://lnkd.in/e2zFw53p #Databricks

11

1 commentaire

Postgres has a highly extensible architecture, but it needs to be modified to support large-scale analytics in order to handle the AI workloads of the future. That was one of the key topics at hand as EDB Senior Director Franck Sidi sat down with EDB Technical Fellow Dr. Marc Linster to discuss the relationship between AI and analytics, as well as how to apply AI techniques to enhance or automate analytics. In order to support large-scale analytics, Postgres has to be extended in these four ways: 1. Decoupling compute and storage - Decoupling the compute engine with the storage engine to allow independent scalability. 2. Dynamic scalability - Horizontal and dynamic scalability for compute with a distributed query engine. The same query can be split and run on multiple computer nodes using a distributed query execution engine. 3. Columnar storage - Support for Columnar Storage format for the tables is essential for query execution speed to access only relevant columns. This also enables efficient data compression as you can apply compression algorithms per column. 4. Vectorization - A vectorization query engine makes it possible to leverage the CPU modern chipset. Vectorization means that you can work on multiple rows in 1 CPU instruction. Single Instruction, Multiple Data (SIMD concept) Want to dive deeper into using Postgres for analytics, search, and AI? Read the full blog here ➡️ https://bit.ly/3ZsbYYL #EDBPostgresAI #JustSolveItWithPostgres #AIAnalytics #LargeScaleAnalytics #PostgreSQL #VectorDatabase

34

1 commentaire

Great post enjoyed the read! I think Data Governance is very important to AI. The quality of the AI’s decisions are tied to the quality of the data presented to the tool. However, I think Data Governance can be taken a step further to even greater enhance AI capability. One of the great limitations around AI is the that it requires integrated data to be powerful, but integrating data is expensive and time consuming, mostly do the requirements of transforming the data. However, there is an approach that leverages data standards, around master data, which Data Governance can help create, that makes datasets directly interoperable at the source. These new standardized data sets can all share data through master data commonality, even though they are de-centralized. So, in this approach, Data Governance not only improves the quality of the data for AI but can be leveraged to increase the variety of enriched data the AI has access to. Enjoyed the post and look forward to more.

2

The Databricks #DataAISummit is here! 🚀 Trying to wrangle big spatial data? Be sure to visit booth #MP3 to learn about our new Analytics Toolbox for Databricks ! While you're there don't miss our talks 👇 https://lnkd.in/eiBy5W6d 🗺

11

𝐇𝐢 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐚𝐛𝐫𝐢𝐜 𝐟𝐚𝐧𝐬 𝐚𝐧𝐝 𝐜𝐮𝐫𝐢𝐨𝐮𝐬 My solutions to perhaps improve this fantastic SAAS transversal Data platform Visit please my last article (study) ==> https://lnkd.in/eNzKARhQ Example : topic #10 and DBT "best" "mandatory ?" support on Fabric <== Over T SQL Warehouses mainly #azure #microsoft #datagovernance #microsoftfabric #bi #data #dataanalytics #datafactory #dataengineering #dbt

1

Data Engineers ! This one's for you: 🧠 The Economics of Generative AI Stephanie Kirmer 🧠 Connecting ML Models in Sagemaker to Snowflake and dbt Hugo Lu 🧠 Introduction to Interpretable Clustering NAKUL UPADHYAY 🧠 Snowflake automatic clustering cost estimation Tim Sander 🧠 Death to Tech Debt? Ari Joury, PhD Check it here; https://lnkd.in/ebdFKNZ5 #dataengineering #dataroundup #startup

5

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 commentaires

⏰ Check out our new blog post : “Chunking for RAG: best practices”. Chunking is a crucial data preprocessing step for RAG systems, but finding the optimal approach can be complex. In this post, we talk about the reasons for chunking data, explore common methods, and highlight Unstructured smart chunking strategies that preserve the semantic structure of your documents. These strategies ensure uninterrupted information flow and enhance retrieval precision, improving your RAG systems’ performance. Key takeaways: - Understanding the importance of chunking for RAG - Common chunking methods - Unstructured's smart chunking strategies - Practical tips for chunking optimization Get started for FREE with our Serverless API and bring your RAG system's performance to the next level. We can't wait to see what you build! Read the full blog post here: https://lnkd.in/eGNvyuJz

57

2 commentaires

Time to 𝗲𝗹𝗲𝘃𝗮𝘁𝗲 𝘆𝗼𝘂𝗿 #DataStrategy! Another weekly review with Aymen series: 𝗗𝗮𝘁𝗮 𝗖𝗼𝗺𝗽𝗮𝘀𝘀:𝗧𝗵𝗲 𝗪𝗲𝗲𝗸𝗹𝘆 #Data & #AI 𝗛𝗼𝗿𝗶𝘇𝗼𝗻! 💥 Discover the latest insights shaping the #Data world. In this week’s selection, let’s talk about: 1️⃣ The last announcement of our partner Databricks, that are open sourcing Unity Catalog: https://lnkd.in/eDNPbFB2 2️⃣ The main challenges in defining data projects: what are they and why data #technology, that is crucial, is also insufficient in its own? https://lnkd.in/eZRN3W_n By Jan Meskens 3️⃣ The #GenAI maturity model. Explore advanced techniques in language model customization: https://lnkd.in/eMA84Bpu By Ali Arsanjani, PhD 📩 Contact us to revolutionize your business! Elodie Trojanowski - Dominique Peiffer - Frank Poireau - Aymen Elloumi 𝗖𝗮𝘁𝗰𝗵 𝘆𝗼𝘂 𝗻𝗲𝘅𝘁 𝗙𝗿𝗶𝗱𝗮𝘆 𝗳𝗼𝗿 𝗻𝗲𝘄 𝗽𝗲𝗿𝘀𝗽𝗲𝗰𝘁𝗶𝘃𝗲𝘀!

8

You can now run pandas on 1TB+ data directly in Snowflake 🐼: 📈 Address painful out of memory issues to scale pandas 🔒 Secure access within Snowflake removes sensitive data risks on local machines 🚀 Accelerate prototype to production and scale without tuning or managing infrastructure Learn more about the general availability of pandas on Snowflake: https://okt.to/Tb5IjF

1 004

26 commentaires

𝐃𝐞𝐚𝐫 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 🔥 Here I come with another PySpark Transformation Challenge 📣 𝐈𝐧𝐟𝐨: You have a DataFrame with two columns: 𝐢𝐝 and 𝐩𝐥𝐚𝐧_𝐝𝐚𝐭𝐞. 𝐘𝐨𝐮𝐫 𝐭𝐚𝐬𝐤 𝐢𝐬 𝐭𝐨 𝐜𝐫𝐞𝐚𝐭𝐞 𝐭𝐡𝐫𝐞𝐞 𝐧𝐞𝐰 𝐜𝐨𝐥𝐮𝐦𝐧𝐬: 1. 𝐈𝐒_𝐋𝐀𝐒𝐓_𝐒𝐈𝐗_𝐌𝐎𝐍𝐓𝐇𝐒: For each id, check whether the date is within the last 6 months compared to the previous row for that id. The first row for each id should always be "YES", and subsequent rows should check if the previous date is less than 6 months before. If it is within 6 months, mark it as "YES", otherwise, mark it as "NO". 2. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐅𝐈𝐍𝐀𝐍𝐂𝐈𝐀𝐋_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current financial year (April 1, 2024 - March 31, 2025). 3. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current calendar year (January 1, 2024 - December 31, 2024). 𝐂𝐫𝐞𝐚𝐭𝐞 𝐒𝐚𝐦𝐩𝐥𝐞 𝐃𝐚𝐭𝐚𝐟𝐫𝐚𝐦𝐞 (Input Data): 𝘥𝘢𝘵𝘢 = [   (1, "2024-08-15"),   (1, "2024-03-10"),   (1, "2023-05-05"),   (2, "2025-02-24"),   (2, "2024-07-15"),   (2, "2024-03-15") ] 𝘤𝘰𝘭𝘶𝘮𝘯𝘴 = ["𝘪𝘥", "𝘱𝘭𝘢𝘯_𝘥𝘢𝘵𝘦"] 𝘥𝘧 = 𝘴𝘱𝘢𝘳𝘬.𝘤𝘳𝘦𝘢𝘵𝘦𝘋𝘢𝘵𝘢𝘍𝘳𝘢𝘮𝘦(𝘥𝘢𝘵𝘢, 𝘤𝘰𝘭𝘶𝘮𝘯𝘴) 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐂𝐨𝐝𝐞? (Expected Data) Let me know the answer in the comments 😀 #pyspark #dataengineering

81

6 commentaires

UPDATE: Release 0.1.2 of onelake-shortcut-tools is out which includes the updated Delta Lake 3.1 features available in Fabric Runtime 1.3 Preview (liquid clustering, default columns) Need to evaluate what external tables can be read from and written to in Fabric? Just install my library via PIP and get a clean report out of compatibility. See blog in comments for how to run it. #deltalake

38

1 commentaire

𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗗𝗮𝘁𝗮 + 𝗔𝗜 With siloed data and complex AI platforms, the challenge for businesses is real: how can we accelerate AI adoption efficiently? The annual State of Data + AI by Databricks is here to answer this and much more. Here are the key insights: 🤖 𝟭𝟭𝘅 𝗺𝗼𝗿𝗲 𝗔𝗜 𝗺𝗼𝗱𝗲𝗹𝘀 𝗶𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻: companies are moving beyond experimentation, deploying more AI models than ever before. 🗣️ 𝗡𝗟𝗣 𝗱𝗼𝗺𝗶𝗻𝗮𝘁𝗲𝘀: Natural Language Processing is the fastest-growing ML application. 📈 𝗩𝗲𝗰𝘁𝗼𝗿 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 𝘀𝘂𝗿𝗴𝗲: 70% of companies using GenAI leverage tools and databases to customise models, with a 377% YoY growth. 🕊️ 𝗢𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗹𝗲𝗮𝗱: 76% of companies using LLMs prefer open-source solutions, striking a balance between cost, performance, and latency. Dive deeper into the report here: https://lnkd.in/dVrKP4hf #DataAI #GenAI #MachineLearning #Innovation #Databricks #DataIntelligence #NLP #AIPlatforms #StateofDataAI

16

2 commentaires

When I read this first, I was like, Tools updated for MySQL 10.1 ? That has not been released yet... https://lnkd.in/e9EDWHWq #mysql

1

Did you miss this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #databricks

7

1 commentaire