Samhar Mahmoud

This weekend I read “Zero-shot Tokenizer Transfer” by Minixhofer et al, 2024. In this paper the authors tackle the limitation of the dependence of language models (LM) to their tokenizer. In their approach, the authors propose to “mitigate” this limitation by changing the original tokenizer to an arbitrary one such that the performance of the LM does not degrade. To do this, the authors train a network that takes the tokenizer as input and predicts the embeddings for a base LM. Once trained, the authors claim that the network generalizes to new tokenizers for different LLMs. To me this paper is interesting because studies the implications of how the units of the vocabulary at hand (tokens) have to the language model. Being able to map across representations seems to be an important computing operation in artificial and biological networks. Paper: https://lnkd.in/gU-6Ja7h

4

👉🏼 Reliability of large language models for advanced head and neck malignancies management: a comparison between ChatGPT 4 and Gemini Advanced 🤓 Andrea Lorenzi 👇🏻 https://lnkd.in/eSW4Admy 🔍 Focus on data insights: - ChatGPT 4 generally outperformed Gemini Advanced in adherence to guidelines and comprehensive treatment planning. - Higher AIPI scores were observed with ChatGPT 4, indicating better overall performance. 💡 Main outcomes and implications: - Both large language models showed potential in aiding multidisciplinary management of head and neck oncology. - Discrepancies in critical areas emphasize the need for further refinement and validation against current clinical standards. 📚 Field significance: - Growing role of AI in enhancing clinical decision-making. - Necessity for continuous updates and validation to fully integrate AI into healthcare practices. 🗄️: [#largeLanguageModels #headAndNeckMalignancies #AIclinicalDecisionMaking]

1

1 Comment

New blog post out by the Bespoke AI team! Hallucination has become a blanket term. This blog post helps clarify and reframe LLM hallucinations in the context of more traditional NLP tasks. We’ve both been thinking this problem for a bit. We’ve been using Provenance-LLM as our primary hallucination detection validator, however we suspect that Minicheck performs better. @bespokelabsai's grounded factuality detector, Bespoke-Minicheck-7B, tops the LLM AggreFact leaderboard. This lightweight model outperforms larger foundation models, including GPT-4 and Mistral-Large 2 on grounded factuality detection. https://lnkd.in/g4tPCAKU

10

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

LLM Explainability... Following on from a previous post about intelligence as an expression of uncertainty, which is possibly a feature, not a bug, of LLMs (per the remarks via Professor Lee), I thought it helpful to post this paper about LLM explainability. It offers a thorough survey of applicable techniques. Explainability, per the paper, can be divided into different approaches according to what is being explained: the fine-tuning paradigm or the prompting paradigm. Furthermore, each paradigm can be broken down into local and global explanations. Local is the explanation of a single prediction where global is an explanation of the model-scale mechanisms. Of course, the particular type of explainability that one might seek is very use-case dependent. In many cases, a natural-language explanation is going to be required, as in getting the LLM to account for its prediction. An obvious example of this is Chain of Thought. Clearly, as LLMs are increasingly deployed in acts of reasoning and decision-science within enterprises, it is important for the human operator to understand the basis of the reasoning. In my research, I have been following the work done to produce inference graphs for the purposes of achieving defeasible reasoning -- i.e. beliefs that can be updated as more (specific) information becomes available. Some work has been done to show that such methods can augment human decision-making, helping, for example, to overcome cognitive biases (of many kinds). In general, one can think of this as a key deployment for aiding decision science -- human wisdom + LLM "observability" -- i.e. because an LLM has a much wider body of knowledge and many more examples to bring to bear upon a given problem. But a human can still "join the dots" better than the LLM in terms of obtuse highly contextualized problem spaces. Given how widespread cognitive biases supposedly are (per the Behavioral Economists' viewpoint), then we ought to expect that base models have inherited these cognitives biases. However, per what Lee said (see my previous post), this could be construed as an advantage in terms of operating in a space of greater creativity. This is perhaps why adaptive sampling methods are so important that allow the predictions to vary in temperature (randomness) according to how much creativity is useful within a given context window of the predicted outputs. Anyway, here is the link to the paper -- https://lnkd.in/gcdKFY5f Enjoy!

11

💡 personal take on 💢 Ai2 Tülu 3, an open-data & open-pipelines LLM. Hugging Face and the open LLM community will benefit a lot from this. CC: Lewis Tunstall, Jeremy Howard, Llion Jones, Adam Treat. 🔍 Post-training reimagined & fully open - Pre-trained language models are powerful but incomplete, lacking in safety, alignment, and specialized capabilities. - Tülu 3 introduces a fully open post-training framework, bridging the gap between open-source and leading proprietary models like OpenAI GPT, Anthropic Claude, and Google Gemini. 🛠️ A suite of tools for the community - Open pipelines: fully transparent post-training methodologies, from data curation to evaluation. - Decontaminated datasets: prepped for skills like coding, reasoning, multilingual interactions, and more. - Evaluation framework: simplifies reproducibility and performance comparison for developers. 🌍 Broad impact, scalable innovation - One can mix and match datasets to tailor a model for your specific use case, without losing its core general capabilities. - Scalable across different model sizes (from 8B to 70B parameters), requiring less compute than you’d expect. - Training from scratch still requires money. - Fine-tuning will get much better quickly, requiring to be only GPU-bootstrapped. 🧠 Cutting-edge techniques: - Reinforcement learning with verifiable rewards, skipping traditional reward models. - On-Policy generations: using a model's outputs to refine its preferences. 💡 Why it matters: Tülu 3 is more of an ecosystem than a model. By releasing all code, data, and infrastructure, Ai2 have further empowers nations, researchers, developers, and entrepreneurs to build models that rival closed alternatives while promoting transparency and collaboration. #ai #aiinnovation #aidevelopment #airesearch #aiforgood #artificialintelligence #chatgpt #datascience #futureofai #genai #generativeai #innovation #languagemodels #llama #machinelearning #mltools #mlops #naturalanguageprocessing #nlp #openscience #opensource #opensourceai #openai #posttraining #tülu3

12

1 Comment

We asked Dvir Ben Or and Andrey Lisovoy what's new and valuable has happened in Data science recently: #xLSTM: Extended Long Short-Term Memory. It aims to overcome some limitations of traditional LSTMs by introducing two main modifications: 👉Exponential gating: Using exponential activation functions for the input and forget gates instead of sigmoid functions. This allows the gates to revise storage decisions more easily. 👉New memory structures: #sLSTM: A scalar memory cell with a new memory mixing technique across cells/heads. #mLSTM: A matrix memory cell that is fully parallelizable with a covariance update rule. These modifications are integrated into residual blocks to create xLSTM blocks, which can then be stacked to form xLSTM architectures. The results demonstrate that xLSTM models outperform other large language models like RWKV-4, Llama, and Mamba on various benchmarks, including next token prediction, common sense reasoning tasks, and fine-grained domain benchmarks across different model sizes. Notably, xLSTM maintains low perplexities even for very long context lengths, excelling at sequence extrapolation. 🐾Our Comment For mobile game development history, this means that xLSTM architectures could potentially lead to more efficient and powerful language models for tasks like game dialogue generation, quest text generation, or any other text-based content creation. The linear computation complexity could make xLSTMs more suitable for on-device deployment in mobile games compared to computational expensive Transformers. #tech #innovation #mobile #game #development

2

"...embeddings enable machine learning models to find similar objects." Read more of Gregory Hodum's article on embeddings https://lnkd.in/eFs6hhAP #ai #softwaredevelopment #embeddings #machinelearning

9

Our episode on the The Data Exchange Podcast with our fantastic host Ben Lorica 罗瑞卡 , and featuring my great colleague Hima Patel , on data preparation for LLM applications (RAG, fine-tuning, instruct-tuning, and pre-training) using the Data Prep Kit is out! Data Prep Kit is the open-source (Apache 2.0 license) framework and transforms for language and code, which we used in IBM Research to prepare the data for the training of #IBM 's #Granite models ( https://lnkd.in/eUvZVK2t ). Running at any-scale, from a laptop to a datacenter and handling from a few pages of text to hundreds of billions of documents, it is available in pure-#Python , #Ray , #Spark and with #KubeFlow pipelines. Feel free to check it out here: https://lnkd.in/ehGjpbzG #IBMResearch #AI #GenAI #LLM

42

2 Comments

Its a jungle out there! 🌿 Take a look at this manuscript mapping similarities/differences in #RWE recommendations across #regulatory and #HTA guidance documents/frameworks. Becaris Publishing Limited #fitforpurpose #RWD #evidence #realworlddata #dataquality #bigdata #regulatoryaffairs #marketaccess #patientjourney #patientoutcomes #clinicaltrials #realworldevidence #drugdevelopment #pharma #FDA #EMA #reimbursement #healthcare #heor #healtheconomics #medicalaffairs #ISPOR #access #ISPE #pharmacoepidemiology

47

👉🏼 Expansive data, extensive model: Investigating discussion topics around LLM through unsupervised machine learning in academic papers and news 🤓 Hae Sun Jung 👇🏻 https://lnkd.in/ef_MF54v 🔍 Focus on data insights: - Utilized data from Web of Science and LexisNexis to explore topic modeling methods for Large Language Models (LLMs). - Evaluated various topic modeling approaches such as LDA, NMF, CTM, and BERTopic based on diversity and coherence metrics. - BERTopic demonstrated superior performance in diversity and coherence across both platforms. 💡 Main outcomes and implications: - News articles covered a wide range of topics related to LLM applications, focusing on specialized domains. - Research papers concentrated on technical aspects of LLM technology. - Insights gained can help understand future challenges and opportunities for LLMs in different sectors. 📚 Field significance: - Enhances understanding of how LLMs are utilized in academic and news contexts. - Provides insights for enterprises leveraging LLMs for service delivery. 🗄️: #topicmodeling #unsupervisedlearning #machinelearning #datainsights

1