John Curry

I'm thrilled to share a bonus episode of the Data Today #podcast on #AIAppreciationDay. This one is particularly special to me: Data and Artificial Neural Networks with Damian Borth.    Neural networks are the backbone of many #AI and machine learning systems. But how can they be applied to real-world scenarios?    I had the incredible opportunity to sit down with Damian Borth, Professor of Artificial Intelligence and Machine Learning at the University of St.Gallen and director of its Computer Science Institute. Damian's ground-breaking work on neural networks spans several crucial areas, including climate change, financial fraud detection, and text-to-speech. It's no wonder he’s a recipient of the prestigious Google Research Scholar Award!    In this episode, we dive into:  What #neuralnetworks are and what they do How their application can help tackle global challenges Insights from Damian's vast experience and groundbreaking research   I found our conversation incredibly enlightening and inspiring, and I can't wait for you all to hear it. Tune in to discover how #ArtificialIntelligence and neural networks could help save the world! #TechForGood https://lnkd.in/ePTYn3AC #MachineLearning #DataScience #Innovation #Research #Data #Technology

44

Responsible AI is not the only thing to focus on. Sustainable AI deployments must follow too.

16

1 Comment

Its a jungle out there! 🌿 Take a look at this manuscript mapping similarities/differences in #RWE recommendations across #regulatory and #HTA guidance documents/frameworks. Becaris Publishing Limited #fitforpurpose #RWD #evidence #realworlddata #dataquality #bigdata #regulatoryaffairs #marketaccess #patientjourney #patientoutcomes #clinicaltrials #realworldevidence #drugdevelopment #pharma #FDA #EMA #reimbursement #healthcare #heor #healtheconomics #medicalaffairs #ISPOR #access #ISPE #pharmacoepidemiology

47

We published a recording of the workshop on data ingestion complexities for maintaining an up-to-date RAG. We covered: 🔸 Basics of data ingestion 🔸 Key challenges in data ingestion 🔸 How dlt addresses these challenges 🔸 Challenges in using vector databases 🔸 Introduction to LanceDB 🔸 Setting up a scalable RAG data pipeline You can find the recording, slides, and a Google Colab notebook here: https://lnkd.in/ejTPFV_9

166

4 Comments

Good article on practical tips to improve the #RAG pipeline - across indexing, retrieval and generation. • Indexing - level up with simple, semantic or #LLM based chunking • Retrieval - implement hybrid search, combining vector and keyword searches • Query Rewriting - optimize queries for better retrieval • Metadata - boost retrieval accuracy with rich contextual data • Autocut - filter out irrelevant information from retrieved content • Reranking - prioritize the most relevant content, enhancing retrieval precision https://lnkd.in/eJMvu3XJ

7

Just released PyLint plugin for Databricks https://lnkd.in/ev_QFZtv It contains 20 different kinds of checks to validate your Python code for best practices and Unity Catalog compatibility. It's an early release, but it already looks at: Apache Airflow DAGs, notebooks, %run cells, leaked PAT tokens, dbutils usage, testability of Apache Spark DataFrame transformations, and more. At the moment, some of the checks are too coarse grained and may produce some false positives, but it's still pretty effective and works on codebases with hundreds of thousands of lines of code. Add it to your CI configurations and get your Python code to the next level.

213

12 Comments

Is your team building AI/ML and struggling to get accurate models into production? Have you ever wondered "Are there better tools out there than what i'm using now? Is there a simpler & faster way to do this?" Imagine building chatbots, AI agents, ML models in 1/10 of the time and effort. Dozens of Fortune 500 companies have, and are glad they found Abacus: The world's first AI super-assistant where AI, not humans, build enterprise-scale applied AI systems and agents. If you're not 100% happy with how you're building AI/ML today, Abacus could be for you too. Ping me if you'd like to have a friendly introduction to learn more! hashtag #AI #ArtificialIntelligence #MachineLearning #MLOps #DataScience #Technology #Innovation #AbacusAI #AIAgents

5

🔝🚨Anomalous emergent behaviors in AI: Hermes 3, has "existential crises" when you use right inputs, a black prompt or ask "Who are you?" or "What do you see?". Here you are some examples of unexpected emergent behaviors at scala. Emergent behaviors are really relevant in AI because thosw appears without especific training and many times only a few times even in the LLM has been trained with the same datasets and procedures. #AI #AGI #LLM #research #artificialIntelligence https://lnkd.in/dyDUEFs4

3

1 Comment

Rewatching my PyCon talk from last year, three things stand out: 1. I was deep into AI-generated graphics back then. In fact, I probably spent more time on prompting than writing the talk! (I'm less into them now; will wait for them to improve.) 2. It's definitely more of an intellectual journey than a business one—fitting for a Python conference. 3. It doesn’t quite follow a structured progression, leaning more autobiographical. But it does reflect the rise of Python in data science and software engineering alongside my own career path.

2

Databricks Unity Catalog is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. With Unity Catalog, organizations can seamlessly govern both structured and unstructured data in any format, as well as machine learning models, notebooks, dashboards and files across any cloud or platform. Review Kiran Sreekumar's blog below on the top 10 Data Governance tips for Unity Catalog! Thanks Kiran Sreekumar! #databricks #unitycatalog #datagovernance

29

Snowflake released the #SnowflakeArctic embed family of models and made them open source under an Apache 2.0 license. The Massive Text Embedding Benchmark (MTEB) Retrieval Leaderboard shows that the Arctic embed model with only 334 million parameters is the only one to perform better than the average retrieval performance of 55.9. The five models, ranging from x-small (xs) to large (l), can be used right away on Hugging Face. Companies use private datasets along with #LLMs as part of a Retrieval Augmented Generation (#RAG) or #semanticsearch service. The impressive embedding models put into practice the technical know-how, search capabilities, and research and development that Snowflake got from Neeva in May. Snowflake Copilot is integrated into Snowflake Cortex, their #AI service, and can be utilized for a variety of tasks such as question answering, and summarization. With these announcements, Snowflake is breaking ground on open LLMs and showing it is continuously innovating for their customers on the AI journey.  

8

The final post in my series on the Unified Star Schema (USS) by Bill Inmon and Francesco Puppini. This time I look at more benefits and come to a conclusion. I have looked at many different data modeling techniques during my career, and none have appealed as much as the USS. The benefits of adopting it are huge, especially for data teams looking to offer self-service access to data. I'm sure there will be edge cases where it doesn't suit, but I think it should be considered very broadly for adoption. It elegantly solves some of the biggest problems in data warehousing, and almost certainly reduces the total cost of ownership of Business Intelligence for any org. Thanks again to Technics Publications who have offered a discount code to buy a copy of the book, which is at the end of the post. https://lnkd.in/g9sumhDm

19

I recently discovered a fascinating aspect of embeddings - they can be clustered effectively! While collaborating on a project, I delved into some research and uncovered this insightful finding. For those interested in exploring this topic further, here's a helpful explanation: https://lnkd.in/gUPHe5E3

7

Interesting approach to data prep for #RAG .. haven’t seen much innovation in this space for a while. This method by Anthropic is still costly for large datasets but the research does shown improvements compared to more classic or hybrid RAG and much cheaper now with catching. Read here: https://lnkd.in/gXxnrWgR #LLM #AI #GenerativeAI #GenAI

The The Data Scientist Magazine, published by my good friend Damien Deighan, is FREE and well worth subscribing to, and not just because he put my ugly mug on the front cover! The next issue is out soon where I've contributed an article on Retrieval Augmented Reasoning (RAR), the answer to the weaknesses in #RAG - gaining context, reasoning logically and providing causal explainability. https://lnkd.in/ePct69XH

43

1 Comment

#RAG or #KnowledgeGraphs⁉️ We have always thought 💭 that it's structured vs unstructured data that drives the choice... Not 🚫 to be #HybridRag Read this paper below 👇 BlackRock NVIDIA "HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction" Abstract: Problem Statement: Extraction and interpretation of intricate information from unstructured text data arising in financial applications, such as earnings call transcripts, present substantial challenges to large language models (LLMs) even using the current best practices to use Retrieval Augmented Generation (RAG) (referred to as VectorRAG techniques which utilize vector databases for information retrieval) due to challenges such as domain specific terminology and complex formats of the documents. Solution: We introduce a novel approach based on a combination, called HybridRAG, of the Knowledge Graphs (KGs) based RAG techniques (called GraphRAG) and VectorRAG techniques to enhance question-answer (Q&A) systems for information extraction from financial documents that is shown to be capable of generating accurate and contextually relevant answers. The Approach: Using experiments on a set of financial earning call transcripts documents which come in the form of Q&A format, and hence provide a natural set of pairs of ground-truth Q&As, we show that HybridRAG which retrieves context from both vector database and KG outperforms both traditional VectorRAG and GraphRAG individually when evaluated at both the retrieval and generation stages in terms of retrieval accuracy and answer generation. The proposed technique has applications beyond the financial domain. #GenAi #Finance

8

3 Comments