Frank Munz ☁️ 🧱

🛠️ Build powerful applications with the integration of #AzureCosmosDB NoSQL and #OpenAI! Sandeep Nair demonstrates how to leverage vector embeddings for seamless multi-modal data operations. Watch now: https://lnkd.in/ezx6s8yu

2

Welcome Howso, a new #validated partner of Databricks! I've been posting about many synthetic data partners in the last few weeks, but Howso is different. Howso’s main differentiator is its focus on #AI_explainability. In industries where trust and compliance are critical (like finance, healthcare, and government), being able to explain how an AI model makes decisions is extremely valuable. In addition, Howso is also focused on #Causal_AI. This helps organizations uncover cause-and-effect relationships, not just correlations, which improves decision-making in complex environments. They uncover the truth about your data. Incredibly happy to have Howso, "The Understandable AI® Platform" as part of the Databricks AI partner ecosystem! Read more here: https://lnkd.in/e-SacuJH John Emmert

44

7 Kommentare

The final post in my series on the Unified Star Schema (USS) by Bill Inmon and Francesco Puppini. This time I look at more benefits and come to a conclusion. I have looked at many different data modeling techniques during my career, and none have appealed as much as the USS. The benefits of adopting it are huge, especially for data teams looking to offer self-service access to data. I'm sure there will be edge cases where it doesn't suit, but I think it should be considered very broadly for adoption. It elegantly solves some of the biggest problems in data warehousing, and almost certainly reduces the total cost of ownership of Business Intelligence for any org. Thanks again to Technics Publications who have offered a discount code to buy a copy of the book, which is at the end of the post. https://lnkd.in/g9sumhDm

19

Snowflake released the #SnowflakeArctic embed family of models and made them open source under an Apache 2.0 license. The Massive Text Embedding Benchmark (MTEB) Retrieval Leaderboard shows that the Arctic embed model with only 334 million parameters is the only one to perform better than the average retrieval performance of 55.9. The five models, ranging from x-small (xs) to large (l), can be used right away on Hugging Face. Companies use private datasets along with #LLMs as part of a Retrieval Augmented Generation (#RAG) or #semanticsearch service. The impressive embedding models put into practice the technical know-how, search capabilities, and research and development that Snowflake got from Neeva in May. Snowflake Copilot is integrated into Snowflake Cortex, their #AI service, and can be utilized for a variety of tasks such as question answering, and summarization. With these announcements, Snowflake is breaking ground on open LLMs and showing it is continuously innovating for their customers on the AI journey.  

8

This gets complex in a hurry as only data scientists seem to be able to stir the pot this quickly. IBM created a new LLM fine-tuning product, InstructLab, on a human feedback loop supported by synthetic data, then open-sourced it and transferred support and licensing over to RedHat. I only found out about this project as it was an afterthought today by a WatsonX dev who mentioned it on an IBM webinar. Mistral Version - https://lnkd.in/gEz8cem9 Llama Version - https://lnkd.in/gJHQC9Py White Paper - https://lnkd.in/ghYS9wsy Fine-tuning a data set is a painful and expensive process, integrating near real-time feedback from subject matter experts or end users is several factors more complex. IBM: "Oh by the way, we have this sitting on a shelf somewhere." #InstructLab #IBM #AI #finetuning #redhat

3

1 Kommentar

Bloom indexes in #PostgreSQL efficiently test for the presence of a set of values, especially useful for multi-column queries where traditional indexes may be less efficient. They are ideal for checking combinations across columns without needing explicit indexes on each. #PostgreSQLPerformance @MinervaDB #dba https://zurl.co/OKDl

5

Questions I get so often it's becoming a broken record... 💡 How can I trust or use GenAI to handle the complexity of medical knowledge, vocabularies, ontologies? 💡 How can I leverage LLMs on-prem vs. API calls to external services, especially for use cases when I am working with PHI? 💡 I’m just starting with GenAI and I am worried about our tech maturity to handle the necessary AI infrastructure to scale. Is there an Easy Button? If you have these questions too - Tune into our webinar with TitanML! https://lnkd.in/eEteyp2j

19

🤔 What are the differences between Data Fabric & Data Mesh? How does Confluent enable these architectures? 🎙️ We answer these questions and more in Episode #4 of our Confluent Capabilities 'Bite Size' podcast series with Italo Nesi and John Dee! Discover how Confluent enhances both of these models through its cloud platform, showcasing it as a versatile solution and demonstrating its crucial role in modern data strategies. 🖥️ Watch the latest episode here: https://lnkd.in/emG9jRJ2 Paul Archer Joanna Grimmond Chris Gallagher Bob Feeney #HybridCloud #MultiCloud #Confluent #DataFabric #DataMesh

10

Thirsty? Building a high-performance #RAG system is like mixing the perfect gin and Tonic. Tonic.ai, a Databricks ISV partner, offers Tonic #Textual, a tool that converts your unstructured data (PDFs, Word Docs, etc.) into AI-ready formats. It integrates with #UnityCatalog, ensuring both accuracy and privacy. Key business benefits: -- Faster decision-making: Tonic Textual preps your unstructured data, delivering faster and more accurate insights. -- Better productivity: Automating data prep saves your teams hours of manual work. -- Improved accuracy: By adding metadata (like names or dates), AI reduces errors and delivers more precise results. -- Seamless integration: Tonic Textual works with Databricks Unity Catalog and Mosaic AI for easy scalability, so your #AI solutions can grow with your business. Wanna do it? Read here for instructions: https://lnkd.in/gAGGNRNT Tomer Benami, Prasad Kona, Eric Peter

73

14 Kommentare

In-memory data processing, similar to #Hadoop and #Spark in the past, provides discontinuity in terms of cost, performance, and flexibility. Our friends at MotherDuck compiled a list of key workflows that leverage DuckDB to revolutionize analytics. We are committed to bringing this innovation to the Data Product landscape through the GoodData Unified Analytics (BI/AI/ML) platform: 🌐 Zero-Copy 🚀 Lightweight Compute ⚡ Pipeline Processing 🖥️ Interactive Data Apps 🔒 Enterprise Security

56

1 Kommentar

Row and column level security is GA in Unity Catalog, letting you apply this across Apache Spark, SQL DW, streaming and AI workloads in a uniform way!

101

5 Kommentare

You can easily deploy any John Snow Labs models within Snowflake's #Snowpark Container Services Ecosystem. This includes 2,500 #Healthcare-specific #NLP and #LLM models, nlp.deploy_model_as_snowflake_udf() will build, tag & push a John Snow Labs model server to your Snowflake image repository and finally create a service & udf from the model and test it. Everything runs within your infrastructure - no data ever leaves your security perimeter and no external API call is made. #ai #generativeai #healthcareai #healthai #datascience #nlproc #textmining

17

🚀 Exciting Announcement from Databricks! 🚀 General Availability of Serverless Compute for Notebooks, Workflows, and Delta Live Tables! 🎉 With this powerful addition, we can now: 🔹 Simplify Infrastructure Management: No more hassles with provisioning and scaling compute resources. 🔹 Boost Productivity: Seamlessly run your data workflows and notebooks without interruptions. 🔹 Optimize Costs: Pay only for what you use with our efficient serverless architecture. This is a game-changer for data engineers, analysts, and scientists who want to focus more on innovation and less on infrastructure. 💡 https://lnkd.in/gt5b_uCF #DataScience #BigData #Serverless #Innovation #DataEngineering #MachineLearning #DataAnalytics #Databricks

4

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Kommentare

SpreadsheetLLM 📈📊 https://lnkd.in/eCwx5aTv "Spreadsheets are characterized by their extensive two-dimensional grids, flexible layouts, and varied formatting options, which pose significant challenges for large language models (LLMs). In response, we introduce SpreadsheetLLM, pioneering an efficient encoding method designed to unleash and optimize LLMs’ powerful understanding and reasoning capability on spreadsheets."

15

3 Kommentare

Really interesting read on the evolution of Data Warehouses, Data Lakes and Lake Houses and how Databricks and Snowflake are battling for the top spot. https://lnkd.in/egg4P5VG This is one of the more complete analysis I've seen on the evolution of the analytics space and it's big gorilla's. In the end, it gets into the AI discussion of big LLM's vs. small models and how SLM's will have a place in the analytics world, which I completely agree with.

9

1 Kommentar

Let's transform raw page view data from Apache Kafka into aggregated insights directly loaded into Databricks Delta tables. 🧱 We explore real-time data aggregation, demonstrating how to downsample page view data across a 10-second time window using just a few SQL queries. By processing data before it reaches Databricks, you can dramatically reduce storage and compute costs while minimizing latency. It's all here 👇 https://lnkd.in/g7QwbSUD

3