Frank Munz ☁️ 🧱

Thirsty? Building a high-performance #RAG system is like mixing the perfect gin and Tonic. Tonic.ai, a Databricks ISV partner, offers Tonic #Textual, a tool that converts your unstructured data (PDFs, Word Docs, etc.) into AI-ready formats. It integrates with #UnityCatalog, ensuring both accuracy and privacy. Key business benefits: -- Faster decision-making: Tonic Textual preps your unstructured data, delivering faster and more accurate insights. -- Better productivity: Automating data prep saves your teams hours of manual work. -- Improved accuracy: By adding metadata (like names or dates), AI reduces errors and delivers more precise results. -- Seamless integration: Tonic Textual works with Databricks Unity Catalog and Mosaic AI for easy scalability, so your #AI solutions can grow with your business. Wanna do it? Read here for instructions: https://lnkd.in/gAGGNRNT Tomer Benami, Prasad Kona, Eric Peter

73

14 Kommentare

I presented about Apache XTable (Incubating) & how it brings interoperability to #lakehouse table formats at Dremio's Subsurface. Check out the recording of the talk in comments! #dataengineering #softwareengineering

48

3 Kommentare

Excited to share my latest blog on #elasticsearch Labs! 🚀 In this post, I dive into an alternative approach for parsing PDFs in #rag workflows, specifically tackling the challenges of extracting and making sense of tables in PDF documents. #openai #vectordb #genai #rag #elasticsearch

66

Patrick Ames from AWS has more experience than just about anyone with managing thousands of Ray clusters and Spark clusters and processing exabytes of data. There are tremendous operational and performance challenges. - managing thousands of clusters - scaling to many petabytes or exabytes - latency and cost The original blog post is here: https://lnkd.in/g-pJhFei https://lnkd.in/gQsCrXPH

101

4 Kommentare

🔥 New Benchmark Blog! - We ran the TPC-DI benchmark across 3 popular Databricks compute platforms to find out what's cheapest and fastest! 👉 Classic Jobs Compute (optimized) 👉 Serverless Jobs Compute 👉 DBSQL Serverless We utilized Databricks's TPC-DI benchmark across 3 different scaling factors, with no code modifications. The results were interesting, here are a few of the "Surprises" ✅ DBSQL was consistently about 2x cheaper than jobs serverless ✅ Optimized classic was up to 5x cheaper than jobs serverless ✅ Optimized classic was up to 2.5x cheaper than DBSQL serverless Check out the details here! Also, feel free to recreate the experiment yourself, all experiment details and code are open source If you need help optimizing your classic jobs compute clusters to lower costs, you know where to find us Sync Computing #dataengineering #databricks https://lnkd.in/eDxv9K8G

35

3 Kommentare

On a recent real user job, Gradient automatically achieved a 53% cost savings on Databricks + Microsoft Azure Jobs with no change in runtime, by auto-switching from Autoscaling to an optimal fixed cluster size. We reported on the performance of Databrick's autoscaling algorithm from a cost efficiency perspective a while ago and describe its challenges (see blog below) While autoscaling may be beneficial, it really depends on the particularities of your job to see if it's truly cost effective. Hopefully the serverless version of autoscaling is more performant as there will be significantly less spin up time penalty. Sync Computing

71

5 Kommentare

There's a big difference between Database Modeling and Semantic Layers- How would you explain it? Even though they look alike, the underlying meaning is vastly different. ▶ The type of data modeling you would do with database modelers like SqlDBM and semantic layers are interrelated in that we help design and organize database structures in a way that supports the creation of a semantic layer. However: ▶ A database modeling solution is focused on the 𝐩𝐡𝐲𝐬𝐢𝐜𝐚𝐥 and 𝐥𝐨𝐠𝐢𝐜𝐚𝐥 aspects of database modeling, ensuring the data structures align with technical requirements and business concepts. ▶ A semantic layer operates at a higher level of abstraction often within 𝐁𝐈 𝐭𝐨𝐨𝐥𝐬 such as PowerBI, Looker, ThoughtSpot, and Tableau. It maps the technical details of data that were designed by tools like SqlDBM or not properly designed at all, to business-friendly terms and concepts. ▶A tool like SqlDBM will essentially lay the 𝐟𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 that a semantic layer translates into an easily navigable and understandable format for end-users. ---- "Enabling non-technical business users to perform data analysis without needing to understand complex database queries or structures. However, leveraging a data modeling tool and having a good data model will make the complexity of building and running a semantic layer more manageable and efficient." - Keith Belanger, 𝘚𝘲𝘭𝘋𝘉𝘔 𝘗𝘳𝘰𝘥𝘶𝘤𝘵 𝘌𝘷𝘢𝘯𝘨𝘦𝘭𝘪𝘴𝘵 & 𝘚𝘯𝘰𝘸𝘧𝘭𝘢𝘬𝘦 𝘋𝘢𝘵𝘢 𝘚𝘶𝘱𝘦𝘳𝘩𝘦𝘳𝘰 ---- This is why you hear someone say "We do our data modeling in Power BI" - they are modeling their 𝐬𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐥𝐚𝐲𝐞𝐫 in Power BI. This is an image from Power BI, if it had Dark Mode it would look even more similar! #DataModeling #DataArchitecture #SqlDBM

55

1 Kommentar

Interesting new open source Event processing platform from Microsoft. https://lnkd.in/gPr4Cifu The features that I am interested in are 1. It’s input schema is modelled on Debezium, another open source CDC product. 2. Enables action on an event 3. Write continuous queries using Cypher query lanaguage Combining such a powerful platform with a LLM would be sone pe suhaga (best of both worlds).

5

1 Kommentar

Fascinating to me how engineering innovation at the tooling level is equally magical to and keeping up with the LLM frontier model innovation! Super exciting progress with the #Azure #AI Model Inference API as a unified API to awesome models from Cohere, Meta, Mistral AI, Core42 and more without having to change orchestration code while integrating with Azure GenAI Gateway within API management for a scalable architecture. https://lnkd.in/ghutPvBp

28

Interesting post by João Salgado about partitioning vs. sharding in #Postgres which lists trade-offs for each. It's great looking at the list of cons for partitioning and seeing that #TimescaleDB actually helps with most of them - Complexity ⬇️ because we transparently manage partition creation and lifecycle for you as data is ingested - Optimization Challenges ⬇️ because we have planner hooks that help with handling data exclusion (you still do need to pay attention to your partitioning key though). - Storage Overhead ⬇️ (although Postgres doesn't really have much overhead for partitions) because you can just choose to compress your data (we often see a 95x reduction in size). Timescale 🚀

18

1 Kommentar

Meta's Llama models have set a new standard for open large language models (LLMs) in the industry. With a remarkable 10-fold increase in downloads from last year, Llama models are leading the way in openness, modifiability, and cost efficiency. This advancement empowers customers to create innovative solutions tailored to their specific needs. Excitingly, Oracle Cloud Infrastructure (OCI) Data Science now fully supports Llama 2, 3, and 3.1 models, even on CPUs. #Meta #LlamaModels #OracleCloud #DataScience

10

🔥 Learn how to go from zero to production-grade ML orchestration on #Azure: https://lnkd.in/gvKKzfBP Thanks to Sachi Desai and Erin Schaffer for your input and feedback; it enabled us to add integrations with Azure Container Insights, AKS node auto-upgrade, and other improvements for Flyte on Azure users!

30

Training-time vs inference-time compute. Sequoia sees generative AI’s market structure to be divided into 3 layers: foundation, reasoning, and application. The foundation layer (foundation models; single-turn “System 1” thinking) is solidifying into 5 or 6 big players with deep enough financial resources to splurge on expensive training-time compute. All others in this space will exit (or are in the process of exiting). The next market evolution has begun in the reasoning layer with the release of OpenAI’s o1 model which adheres to a new scaling law. This layer provides multi-turn “System 2” thinking and agentic capabilities by trading off training-time with inference-time compute (OpenAI calls it test-time compute). It will likely be dominated by the same big players. But software developers of the world should rejoice. The time has come for them to shine in the application layer where software and enduring value can be built using upcoming cheap/fast foundation and reasoning layer services. These software services will re-imagine and re-tool the trillion-dollar services market. https://lnkd.in/g3muU6iF

22

3 Kommentare

🔺 #AIDataLineage : 🔻 is the ability to trace the complete lifecycle of data and experiments with the data used in #AI #model development and deployment. This is a make or break capability for successfully adopting AI for 📈 business impact 📈as organizations that attempt the daunting task of training, tuning and testing models; often domain-specific models for their industries. This complete picture is crucial for ensuring : ✴ reproducibility of experiments, ✴ understanding the reasons or ✴ clues to failures or poor outcomes, ✴ maintaining data quality necessary to support AI workloads, ✴ fostering collaboration between data and ML teams, and ✴ externally, being able to adhere to regulatory compliance standards and audits. Google Cloud's integrated suite of services, comprising #VertexAI Pipelines, #BigQuery, and #Dataplex #DataCatalog, provide a robust integrated framework for managing and tracking this intricate data journey. This article delves into the intricacies of this integration, highlighting its profound benefits for research scientists and data practitioners alike. Gerrit Kazmaier Warren Barkley Nenshad Bardoliwalla Andi Gutmans Hamidou Dia

39

Join PyTorch at Snowflake's #DataCloudDevDay on June 6 in San Francisco to learn how to accelerate GenAI performance with Native PyTorch. Register: https://hubs.la/Q02xQStt0

18

New Blog: Using Apache Hudi & Iceberg tables in Databricks with Apache XTable (Incubating) 🎉 In this hands-on blog, I show how we can use XTable to interoperate among the 3 open table formats within Databricks environment. I explore a use case where a team of data scientists run ML experiments using MLflow on top of Hudi/Iceberg tables by consuming these tables as Delta Lake. Some key points to consider & how XTable helps: ✅ While predominantly focused on Delta tables, this team also would like to access Apache Hudi/Iceberg datasets from other teams to enhance their ML models. ✅ In the past, the data platform team has either duplicated these datasets to rewrite them as Delta Lake or conducted one-off migrations (costly). ✅ Data Platform team's goal: the ability to read table formats other than Delta, should be seamless & dependency-free, without the need for additional configurations. ✅ Irrespective of whether the source table is Hudi, Delta Lake or #ApacheIceberg, users should just be able to read the dataset as if it were a Delta Lake table using spark.read.format(“delta”).load(“table_name”) Link in comments. #dataengineering #softwareengineering

123

6 Kommentare

Federated governance is crucial to a successful Medallion Mesh approach to scalable data platform evolution. Check out this practical guide to Catalog layout from Karthik Subbarao https://lnkd.in/gWJzKRfq

91

1 Kommentar

The DataFusion SIGMOD paper is officially out. Congrats to Andrew Lamb and the other authors! The paper is great, accessible read covering the architecture of a modern, state-of-the-art query engine. Also exciting to see Arroyo (YC W23) get a mention! The coolest thing about the paper (and about DataFusion itself) is that it demonstrates how building the engine in an open, extensible way doesn't preclude great performance. Systems built on top of DF (like Arroyo) can easily add new functions, operators, tree-rewrites, optimizations, data types, and more while using the the same tools and infrastructure that the built-in functions/operators/etc. are built with. This is incredibly powerful, and has already led to a blossoming of powerful data systems including Comet (a Spark accelerator), LanceDB (a vector DB and query engine), and ParadeDB (which embeds an OLAP DB in postgres). Read the paper here: https://lnkd.in/dpknuPy4

33

2 Kommentare

Blogged: Interoperability between lakehouse table formats with Apache XTable (Incubating). Scenario: Dremio -> Apache Hudi ❌ Dremio -> Apache Iceberg ✅ There are 2 analytics teams in an organization. Team A uses Hudi to manage some of their most critical, low-latency data pipelines. Team B focuses on ad hoc analysis, BI, and reporting, using Dremio's compute engine with Iceberg. ❌ Challenge: Team B wants to do analytics using data from the 2 different datasets within the organization. To do so, Team B has to use the dataset generated by Team A (stored as a Hudi table) and combine it with their dataset (the Iceberg table) ✅ Solution: Interoperability between the formats with Apache XTable. Detailed reading link in comments. #dataengineering #softwareengineering

105

4 Kommentare