Neil McNaughton’s Post

Why Evaluate RAG Systems? HIGHLIGHTS Ragas evaluates RAG systems' effectiveness. It offers specialized metrics for assessment. Crucial for refining language model quality. #RAG Learn More: https://lnkd.in/ddH2WyEe

Excited to share that my latest paper, Enhancing Multi-step Brent Oil Price Forecasting with Ensemble Multi-scenario Bi-GRU Networks , has been published!

How do you evaluate your RAG system? Not only RAG, but how do you evaluate the results generated using an LLM? There are metrics like BLEU and ROUGE, but they have their limitations. For example, BLEU compares the overlap of tokens between predictions and references, rather than assessing meaning. This can lead to discrepancies between BLEU scores and human ratings. To assess complex systems built using LLMs, we need multiple metrics that target different aspects of the system. For instance, retrieval and generation require different evaluation metrics. This brings us to a framework that addresses these challenges: RAGAS. RAGAS (RAG Assessment) is a framework designed to evaluate your Retrieval-Augmented Generation (RAG) pipelines. RAG refers to a class of LLM applications that use external data to augment the LLM’s context. While there are existing tools and frameworks for building these pipelines, evaluating and quantifying their performance can be challenging. This is where RAGAS comes in. In this article, I discuss how to get started with RAGAS, including various evaluation metrics and a specific focus on summarization scores. If you have any additional insights or knowledge on this topic, please feel free to share!

While reading about Answer Semantic Similarity ( a metrics for measuring RAG output in ragas library), I came upon this paper. Description and Evaluation of Semantic similarity Measures Approaches If you're interested in this field ( evaluation of LLM/Models etc), this paper is a good read. https://lnkd.in/g7cgkain #modeleval #modeltesting

In case you wonder what AI can accomplish, I tried something that will probably be appreciated by all those who love to listen to podcasts and maybe do not even have the time to read scientific publications or other longer documents. I submitted a pdf of a manuscript to be converted to a podcast like discussion with no further input than just the pdf file of our latest publication (https://lnkd.in/evexY6VX). Looking forward to hearing (or reading) your impressions. https://lnkd.in/e2NZr-Pf Disclaimer: I tried it with various articles (pdf files), some of these were summarized well, others came across rather superficial or even seemed to be just the cliffhanger but not the core message to be conveyed by the generated dialogue. So indeed I still encourage you to actually read lengthy documents, as they contain definitely much more content than can be reasonably squeezed into a few minutes of dialogue.

Curious about what RAGAS is for RAG in LLMs? Check out the detailed guide in the link below: RAGAS for RAG in LLMs - A Comprehensive Guide to Evaluation Metrics "https://lnkd.in/dnuhg_Ej" #llm #genai

New to Low-Rank Adaptation (LoRA) and other parameter-efficient fine-tuning approaches? Don't miss Cameron R. Wolfe, Ph.D.'s comprehensive guide, which covers both their underlying theory and practical considerations.

[one day one sharing] 100%: takeaways are RAG system evaluation using: Normalized Discounted Cumulative Gain(NDCG) vs MRR (Mean Reciprocal Rank) https://lnkd.in/gRxSmeNZ & using ragas: llm-as-a-judge

I am excited to announce the online publication of our recent research, "Learning-Based Model Reduction and Predictive Control of an Ammonia Synthesis Process." In this work, we propose a complex multi-scale model intended for real-time dynamic optimization of the ammonia synthesis process. The proposed model reduction framework benefits from a time-series perspective and LSTM-based encoder-decoders to dynamically represent the process in model predictive control. This approach showcases an efficient integration of multi-scale models with AI-based surrogates for the dynamic optimization of the ammonia synthesis process system. I am incredibly grateful to my advisor Davood B. Pourkargar and my colleague Thiago Oliveira Cabral for their support and contributions to this work. I am looking forward to the further conversation that this research would initiate. You can read the full paper here: https://lnkd.in/gxKVeguA #Research #AmmoniaSynthesis #PredictiveControl #ModelReduction #AI #TimeSeries #ProcessSystemEngineering