Vid Kocijan

👨‍🔬 🚀 Thrilled to announce the new paper, “Softening Ontological Reasoning with Large Language Models” co-authored by our scientists! 🔗 https://lnkd.in/drX-vKyQ 🔍 Why It Matters: Prometheux data platform enables highly optimised big data queries to fragmented data sources and schemas, showing exactly how it obtains results. 💡This work will advance Prometheux integration with LLMs, further complementing the precision of structured data-processing with the flexibility of language models, empowering organisations to make real-time, fully auditable decisions driven by both structured and unstructured data. #ExplainableAI #knowledgegraphs #AI #neurosymbolic #LLMs

41

5 Comments

Are you facing difficulties in deploying LLMs? Garvan D., a Technical Staff Member at Anthropic, has crafted an insightful guide on deploying Large Language Models (LLMs). This guide addresses the challenges and solutions in LLM deployment. In this walkthrough he builds a simple classifier and shares prompt engineering techniques to improve the model's accuracy.

1

The development of retrieval-augmented generation techniques has grown considerably in the area of large language models and gen Al application development. However, there is an inherent need for a standardised framework for comparing and evaluating RAG approaches in a compatible environment. LLM Orchestration frameworks such as Llamalndex and LangChain offer a huge set of customisations when it comes to RAG, but often fail in meeting the needs of researchers due to their flexibility. FlashRAG offers an open-source modular toolkit that helps researchers compare and reproduce existing RAG methods within a single unified framework. This article explores FlashRAG and its internal architecture towards achieving modularity and RAG reproducibility. Read More 🔗 https://lnkd.in/dNxmSSUH

2

🚀Research Paper Highlights: "Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information" by Ming Jiang et al. Large language models (LLMs) have garnered significant attention for their impressive performance in complex reasoning tasks. However, new studies indicate that their reasoning abilities can be greatly reduced when problem descriptions include irrelevant information, even when advanced prompting techniques are used. This paper addresses this issue by introducing Analysis to Filtration Prompting (ATF) method. 🔍 Technical Approach: GSMIR Dataset: A new dataset crafted with primary school math problems that include misleading irrelevant information to test and enhance LLMs' reasoning skills. ATF Method: Proposes the Analysis to Filtration Prompting (ATF) method to help LLMs identify and filter out irrelevant information during reasoning tasks. 📊 Methods and Techniques: Irrelevant Information Design: GSMIR contains thematically relevant but logically misleading details, mimicking real-world interference. Prompting Techniques: Evaluates prompting methods like Chain-of-Thought (COT) and Zero-shot COT on GSMIR to understand their sensitivity to irrelevant information. 📈 Evaluation: Performance Enhancement: ATF significantly boosts LLMs' reasoning accuracy when dealing with irrelevant information on the GSMIR dataset. 🏆 Results and Comparisons: LLM Limitations: While LLMs can detect irrelevant information, they struggle to filter it out, affecting reasoning performance. ATF Success: The ATF method effectively reduces the impact of irrelevant information, making LLMs more robust across different prompting techniques. ⚠️ Limitations: Residual Challenges: LLMs still face difficulties in fully excluding irrelevant information, even with the ATF method. Dataset Constraints: While challenging, the GSMIR dataset doesn't fully capture the complexity of real-world scenarios. 🔗 Further Reading: https://lnkd.in/eXMbBFJ9 🌟 Stay tuned for more updates on innovative methods and advancements in AI! #MachineLearning #AIResearch #LanguageModels #innovation

5

In this Technical Deep Dive, we share our approaches in using Large Language Models (#LLMs) to make numerical-type data(sets) like DHS, JMP, HDX more accessible with the example of WASH data. In this 15 min. video Olivier Mills and team cover: 1️⃣ Basic scripting to extract core data from datasets from API endpoints 2️⃣ How to use LLMs to generate LLM-friendly schema definitions 3️⃣ How to structure a AI-friendly database store that can scale to thousands of datasets and millions of data indicators 4️⃣ How to use vector querying to retrieve relevant indicators from the store based off of a natural language (english) query 5️⃣ How to build a prompt that generates valid SQL query using LLMs with the above retrieved data schema. The various components form a single AI system combining data ingestion and data exploration using a software layer, a data layer and multiple LLM layers. ⏭️ Next steps will be to expand the datasets, automate other data source ingestion and further test the system with more complex plain english queries. We plan to incorporate this as a new feature for WASH AI after testing. Feel free to reach out if you would like a copy of the Notebooks, sample prompts and data schema definitions. We are #opensource The initial work was done in collaboration with Data for Good YYC (see https://lnkd.in/g3AijS93)

3

1 Comment

🌟Announcing ROLEBENCH, a framework designed to evaluate the performance of Role-Prompting across various datasets and Large Language Models (LLMs). The results across some benchmark datasets demonstrates better performance than results mentioned in model cards [Refer to Github Repository for results]. A free-tier colab notebook to test ROLEBENCH on Llama 3-8B: https://lnkd.in/g4zDRHZf Role-Prompting assigns specific roles to the AI, such as acting as a teacher, a historian, or a technical expert, to see how well the model can adapt its responses based on these roles. ROLEBENCH assess how well different LLMs perform when given these role-specific prompts. This tool will offer valuable insights into the capabilities and limitations of current LLMs. ROLEBENCH currently supports Llama 3-8B, Phi-3-mini, Mistral-7B and Gemma-7B across some benchmark datasets. Github : https://lnkd.in/gbsax-hX #AI #MachineLearning #RolePrompting #AIResearch #LLM #DataScience

80

4 Comments

Interesting paper about the connection between LLMs and structural semantics https://lnkd.in/gDPqd9Sq

35

1 Comment

A Survey of RAG and RAU: Advancing Natural Language Processing with Retrieval-Augmented Language Models Quick read: https://lnkd.in/gnGqk44W Paper: https://lnkd.in/g8-VftbX #llms #largelanguagemodels #ai

20

Researchers at Google released Gemma Scope, a tool that reveals how each layer in Gemma 2-family large language models responds to input. By using over 400 sparse autoencoders (SAEs) to interpret embeddings from different layers, Gemma Scope probes how distinct concepts are represented within the model. This approach could help answer fundamental questions about how language models process information, adapt through fine-tuning, and handle different types of prompting. Read our summary of the paper in #TheBatch: https://hubs.la/Q02PHtlZ0

155

3 Comments

For all large language models (LLM) can do, it's important to recognise what they can't. This research paper addresses a critical question: How reliable is LLM's knowledge? The authors, including the Head of Aveni Labs, Dr. Alexandra Birch, and Dr. Barry Haddow, our Head of Natural Language Processing, introduce a novel metric called MOdel kNowledge relIabiliTy scORe (MONITOR) to assess LLM's factual reliability. MONITOR works by asking models the same question in different ways and seeing how likely the model is to give the same answer each time. This innovative approach helps identify situations where an LLM might be making things up rather than providing facts. Read more: https://hubs.la/Q02w7fJk0 #LLM #largelanguagemodels #naturalLanguageProcessing #machinelearning #artificalintelligence

5

1 Comment

Large Language Models as Markov Chains (October 2024) Paper: https://lnkd.in/gwE6sC73 Abstract: "Large language models (LLMs) have proven to be remarkably efficient, both across a wide range of natural language processing tasks and well beyond them. However, a comprehensive theoretical analysis of the origins of their impressive performance remains elusive. In this paper, we approach this challenging task by drawing an equivalence between generic autoregressive language models with vocabulary of size T and context window of size K and Markov chains defined on a finite state space of size O(TK). We derive several surprising findings related to the existence of a stationary distribution of Markov chains that capture the inference power of LLMs, their speed of convergence to it, and the influence of the temperature on the latter. We then prove pre-training and in-context generalization bounds and show how the drawn equivalence allows us to enrich their interpretation. Finally, we illustrate our theoretical guarantees with experiments on several recent LLMs to highlight how they capture the behavior observed in practice."

1

📄 NEW RESEARCH BLOG! In Simon Prince's latest work, Bayesian Neural Networks, he explores Bayesian learning for neural networks from the parameter perspective. This piece is part VI in a series where machine learning is considered from different viewpoints. 🧑‍💻 You can read the blog now at: https://lnkd.in/gnUhDXiF 🔎 Missed parts I-V? Find them here: https://lnkd.in/gFA2cEC9 #ML #machinelearning #AI #AIinCanada #research #researchtutorial #neuralnetworks #Bayesianmethods

33

Part two of our educational research study series, “From neural nets to agent frameworks - techniques and tools for solving your NLP problem” is now up on our blog. In this study, we explore the Hugging Face transformers library to see how transformer model architectures perform on our text classification problem. As a reminder, this series is aimed at software developers and therefore will be technical in nature. https://lnkd.in/gTCUXi3h

1

An Open-Source Language Model for Evaluating LLMs Researchers have introduced Prometheus 2, a powerful open-source language model specifically designed to evaluate the quality of responses from other language models. This addresses the limitations of proprietary evaluator LMs like GPT-4, offering a transparent, controllable, and affordable alternative. Prometheus 2 surpasses existing open evaluator LMs by: - Achieving high correlation with human judgments and proprietary LMs: Scores closely mirror human evaluations and those of models like GPT-4 and Claude. - Handling both direct assessment and pairwise ranking: Offers flexibility for different evaluation schemes. - Evaluating based on custom criteria: Goes beyond general qualities like helpfulness and harmlessness, allowing for fine-grained assessment tailored to specific needs. Key advancements in Prometheus 2: - Preference Collection: A new dataset with pairwise ranking feedback and 1,000 custom evaluation criteria. - Weight merging: Combines models trained on direct assessment and pairwise ranking data, resulting in a unified evaluator LM that excels in both formats. Prometheus 2 represents a significant step towards a more reliable and accessible evaluation of LLMs, paving the way for further advancements in the field. Read the full paper: https://lnkd.in/dU7sit-Y #Prometheus2 #LLMs #Evaluation #OpenSource #AI ✔️ Click "Follow" on the Cohorte page for daily AI engineering news. Credits: Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo

8

1 Comment

Recently, our colleague Łukasz Łaszczuk showcased our proof of concept on integrating LLMs with Knowledge Graphs using a human-in-the-loop agent. DataWalk's Knowledge Graph feeds LLMs the right data, boosting reliability and reducing hallucinations. This method is perfect for advanced AI applications where RAG and GraphRAG are insufficient. #AI #LLM #KnowledgeGraphs

53

🚀 Introducing LibMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models 🚀 Today, we're thrilled to unveil LibMoE a powerful, modular framework designed to streamline Mixture of Experts (MoE) research, training, and evaluation! 🌟 Why LibMoE? Built on three core principles: - Modular Design - Efficient Training - Comprehensive Evaluation LibMoE makes large language models with MoE architectures more accessible for researchers by standardizing training and evaluation pipelines, helping you unlock the potential of MoE like never before. Dive in, explore, and elevate your research! 🔗 GitHub: https://lnkd.in/g3ksfsa8 🔗 Hugging Face: https://lnkd.in/gG_credQ 🔗 Paper: https://lnkd.in/g8pYeeP7 Other research that collaborates with me, particularly around CodeMMLU, is also interesting. See more at: https://lnkd.in/gWeTf8mg.

34

Here's a snippet demonstrating how you can use Mirascope to make your LLMs take conditional actions based on the classification of an input. Check out the full example at https://lnkd.in/eV8wE5C2

11

2 Comments

The Power of Attention Filtering for Robust Reasoning in Large Language Models A recent study published on Arxiv explored a critical challenge in large language models (LLMs) - the impact of irrelevant information on their reasoning abilities. The paper proposes a novel method called Attention Filtering Transformation (ATF) to address this issue. ATF tackles the problem in two stages: first, it identifies irrelevant information within a given prompt, and then it filters it out to improve the LLM's focus on the core elements necessary for accurate reasoning. This research has significant implications for the development of more robust and reliable LLMs. By mitigating the influence of irrelevant details, ATF paves the way for LLMs to deliver superior reasoning performance across various tasks. It's important to remember that, like any machine learning technique, the effectiveness of ATF can vary depending on the specific problem and dataset. Further research is needed to explore how ATF can be adapted and optimized for different use cases. #ai #llm #prompt * https://lnkd.in/grre_GJH

7

There are lots of AI project charts out there with tiny logos that fit nicely into tiny boxes of functionality. Then it's only you to connect all the tiny logos together into a working system. What if there is an all-in-one solution? https://lnkd.in/dxWDeey

6