Dirk-Jan van Veen

RAGEval - Novel framework for automatically generating LLM evaluation datasets Limitations of existing RAG benchmarks Current RAG benchmarks primarily assess LLMs' ability to answer general knowledge questions. They don't effectively evaluate RAG systems' performance across various vertical domains. RAGEval Framework RAGEval is a new framework for automatically generating LLM evaluation datasets. The RAGEval framework operates by summarizing a schema from seed documents and applying configurations to generate diverse documents. It then constructs question-answer pairs based on both the generated articles and the configurations used. RAGEval also introduces three new metrics: Completeness, Hallucination, and Irrelevance. These metrics are designed to carefully evaluate the responses generated by LLMs, offering a more comprehensive assessment. Benefits of RAGEval RAGEval enables better evaluation of LLMs' knowledge usage ability in vertical domains. It helps distinguish between knowledge sources in question answering, differentiating between parameterized memory and retrieval. #rag #llms #generativeai #llmevaluation

106

1 Comment

Get started with the most powerful LLM library: AdalFlow--The PyTorch library to build, eval, and auto-optimize any LLM tasks. Though only public for two months, AdalFlow has been widely adopted by tech companies in Silicon Valley, including LLM engineers from Meta and various top tech startups. 1️⃣ Create your first chatbot, RAG, and Agent—all in one Colab notebook https://lnkd.in/g23YDaHt 2️⃣ Deep dive on Generator https://lnkd.in/dCRhXg2U 3️⃣ LLM evaluation playbook https://lnkd.in/gTvD93mP 4️⃣ Get started with end-to-end Auto-optimization -- Question Answering: https://lnkd.in/gtECHuGi -- Classification: https://lnkd.in/g4py2e6w 👉 Github repo: https://lnkd.in/g-KN9-dV Happy building! 😊 #adalflow #artificialintelligence #machinelearning #llms

137

2 Comments

Kolmogorov Arnold Networks were supposed to be Machine Learning 2.0, but where are they in R&D world right now? And why was the idea of KAN so big and groundbreaking? The simplest(and the actual) difference between KAN and MLP based Neural Net would be that MLP trains weights and biases to fit your data, while KAN trains the activation function to do so. KAN achieves this feat by using a cool activation function called B spline and trains the control points in it. B-spline reminds me of my early days in computing where I used to work at my mom’s photocopy center and do tasks on photoshop. There is a tool in photoshop where you’d use multiple points to control the curvature of the selection to fit perfectly around a non straight object. B spline is a similar concept where you’d use multiple points to get a curve that is most desirable for your use case; in this case it would be to maximize the impact of each input to the final output. The recent KAN paper made bold claims about its efficiency over MLP and provided a well detailed flowchart on when to use KAN vs other alternatives. The efficiency shown was a result of training features having its own localized activation function, which greatly reduces the amount of complexity required to capture its behavior. While most results on real data about this exciting piece of technology under construction, researchers need more conclusive evidence to adopt KAN as widely as it was supposed to be. The future of KAN definitely looks exciting to me. As I deeply resonate with AVB, In a world where advances in AI is adding one additional layer (and some niche functions) in LLMs, seeing a fundamental level of change in the way we think about Neural Networks is exciting. I call for research in establishing KAN’s useability in the near future. If you liked this post and are still reading, do check out: Me: https://lnkd.in/gK3TJ4Db Further Reading: https://lnkd.in/g2-nbRtr Video Reference: https://lnkd.in/gxTptrkc PS: This is my sixth post on the series of 52 posts I would make throughout the year to explain some of the cool stuff I’ve been working with Nick Rahimi and Saydul Akbar Murad in the Cyber Innovation Labs at University of Southern Mississippi.USM School of Computing Sciences and Computer Engineering

56

7 Comments

🧠 Available Tutorial Material: Self-supervised denoising 🧠 Earlier this summer, I gave a tutorial on self-supervised denoising methods for seismic processing. As a strong believer in open science, I’m happy to share this material with the wider community! Check out our Self-Supervised Denoising Tutorials (https://lnkd.in/e5RXMs-j). This collection of Jupyter notebooks covers denoising techniques aimed at reducing random and coherent noise, with no labelled data needed. It also showcases explainable AI for designing optimal masks, demonstrated on synthetic and field data. Although intended primarily as a teaching resource, the material can also be a good starting point for those intending to develop denoising algorithms and/or to benchmark such algorithms. We’d love to know how the tutorial helps in your work - feel free to reach out and share your progress! As always, I must give a special thank you to my co-authors Matteo Ravasi and Sixiu Liu, who worked with me on developing these algorithms. #OpenScience #AI #SelfSupervisedLearning #Denoising #XAI #MachineLearning #Collaboration

123

1 Comment

Generic LLMs lack the depth when it comes to specific domain understanding hence it’s important to do domain adoption of these LLMS especially when used for niche domains and use cases. There are 3 broad ways to do Domain Adaption of these LLMs. 📌 Domain Specific Pre-training: When you pre-train a model for a specialised domain using extensive domain specific training data. Eg : BloombergGPT for finance 📌 Domain specific Fine tuning:  Adapting a pre-trained LLM for specific tasks or domains. Eg : ChatDoctor , fine-tuned on LLAMA using medical data 📌 RAG (Retrieval Augmented Generation): In this approach, you do not train a model, instead enhance the response quality from LLMs by incorporating up-to-date and relevant information for external data sources. Linking a few papers, if you want to deep dive into any of these topics. #llm #ai #genai #nlp

43

7 Comments