Aniket Maurya’s Post

Ragas v0.2 - LLM Application Evaluation Ragas v0.2 is your ultimate toolkit for evaluating and optimizing LLM applications. <<<Key Features>>> 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics. 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios. 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools. 📊 Build feedback loops: Leverage production data to continually improve your LLM applications. #ragas #llms #evaluation #nlproc

I’m thrilled to share a new series of models I open sourced today called “Self-RAG Classifiers.” This work draws inspiration from the paper "Self-RAG" (as the name implies.) Self-RAG fine-tunes a single language model to generate reflection tokens. I've flipped the problem and developed a series of models that generate these tokens. The core LLM handles the most challenging task of generation. Separate classifiers handle reflection. This approach has the benefits of Self-RAG with more flexibility. Every RAG system will work best with different foundation models. This approach makes it easy to hot-swap the model responsible for generation and still generate reflections. This also stands for the individual classifiers. If you want to adapt one to your system you do not need to retrain the entire model, only the single classifier. I'm currently benchmarking the system performance for RAG. Stay tuned for updates! #largelanguagemodel #artificialintelligence #retrievalaugmentedgeneration Original Paper: https://lnkd.in/d5BGer4h Hugging Face collection: https://lnkd.in/djyxE3bj

What is stopping you from adopting LLM use cases and other AIML models? How do you trust these models and how do you calibrate the trust of end users and other stakeholders including regulators? Why do we need new tests, additional evaluation metrics and enhanced validation framework to manage the model risk associated with them? I will try explaining all of those and say Explainability is all you need. you can find more detail about the event on Chartis Research and signup to attend on the link: https://lnkd.in/eTfdMHdE

Draper’s Sam Lasser recently presented at the 2024 Binary Analysis Research Workshop on a paper he co-authored on the Comparative Binary Analysis Tool (CBAT). We asked him a few questions below: What is CBAT?: “Software engineers sometimes “patch” programs to fix bugs or add new features. Patching is easier when you have access to the program’s original source code, but sometimes, all you have is binary code—the sequence of zeros and ones that a computer executes. CBAT is a tool for checking the correctness of binary code transformations. The tool compares the pre- and post-patched versions of a binary program to confirm that all (and only) the intended changes were made.” What do you hope for the future of CBAT?: “I hope that a broad range of users will try CBAT and suggest new features, or create those features themselves.” READ THE PAPER: https://lnkd.in/eGPXcb74

An LLM evaluation framework is a software package that is designed to evaluate and test outputs of LLM systems on a range of different criteria. The the performance of an LLM system (which can just be the LLM itself) on different criteria is quantified by LLM evaluation metrics, which uses different scoring methods depending on the task at hand. #llms #llmops #aiml #gemini #chatgpt4 #mlops #ml https://lnkd.in/dHruRAxv

Retrieval Many LLM applications require user-specific data that is not part of the model's training set. The primary way of accomplishing this is through Retrieval Augmented Generation (RAG). In this process, external data is retrieved and then passed to the LLM when doing the generation step. LangChain provides all the building blocks for RAG applications - from simple to complex. This section of the documentation covers everything related to the retrieval step - e.g. the fetching of the data. Although this sounds simple, it can be subtly complex. This encompasses several key modules. #RAG #retriver

The latency in the feedback signal of WADC can severely affect its control performance. In the evolving data-driven control techniques, the quality of feedback signals becomes even more critical. In addition, the variable delay and packet drops encountered in feedback signals pose significant challenges to obtaining a good control performance. We are happy to share our recent publication, in which we proposed new compensation techniques for variable delay and package drop. https://lnkd.in/ds7qJ3VC

This aims to provide clarity and conciseness to nodes and other interested parties regarding the new rules and their implications. #BSV #BSVblockchain https://lnkd.in/gVYMbnMm