Jos Gheerardyn

Complex Problem Solving can be addressed using advanced data analysis techniques, like those based on knowledge graphs. Julian Shun of MIT has made significant strides in developing parallel algorithms for processing large-scale networks efficiently. His research focuses on high-performance graph algorithms that accelerate complex data processing, essential for applications like fraud detection and personalized recommendations. His work on dynamic algorithms further enhances our ability to manage vast, evolving datasets, providing timely and reliable insights. Learn more about this here: https://lnkd.in/dakRQNiZ

23

Excited to share insights on the paper 𝘅𝗟𝗦𝗧𝗠: 𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱 𝗟𝗼𝗻𝗴 𝗦𝗵𝗼𝗿𝘁-𝗧𝗲𝗿𝗺 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗕𝗲𝗰𝗸 𝗲𝘁 𝗮𝗹., 𝟮𝟬𝟮𝟰) that I presented to my colleagues last week — a potential new direction for language and time-series models! 🚀 𝗪𝗵𝗮𝘁 𝗜𝘀 𝘅𝗟𝗦𝗧𝗠? 💡 In simple terms, xLSTM is an advanced neural network architecture meant to better understand and predict sequences, like sentences or time-series data. It builds upon traditional LSTM models by introducing new ways to remember important information and forget irrelevant details, leading to more accurate and efficient learning. 𝗪𝗵𝗮𝘁'𝘀 𝗡𝗲𝘄 𝗕𝗲𝗵𝗶𝗻𝗱 𝗧𝗵𝗲 𝘅𝗟𝗦𝗧𝗠? 🧠 𝗧𝘄𝗼 𝗠𝗲𝗺𝗼𝗿𝘆 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝘀, 𝗢𝗻𝗲 𝗦𝗲𝗾𝘂𝗲𝗻𝘁𝗶𝗮𝗹 𝗔𝗻𝗱 𝗢𝗻𝗲 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: • 𝘀𝗟𝗦𝗧𝗠: Nearly identical to LSTM, but introduces new connections between the components, as well as exponential activation functions. The new connections are referred to as memory mixing and improve the ability to revise stored information, while the exponential gates allow for more dynamic memory updates. • 𝗺𝗟𝗦𝗧𝗠: Uses a matrix memory cell with a query / key / value approach to information storage and retrieval (similarly to self-attention), increasing capacity and enabling full parallelisation. 𝗛𝗼𝘄 𝗜𝘁 𝗪𝗼𝗿𝗸𝘀: 🧩 xLSTM's architecture is a stack of sLSTM and mLSTM blocks, enhancing the model's ability to learn from sequences and handle long-term dependencies more efficiently. 𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗜𝗺𝗽𝗮𝗰𝘁: 📈 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗚𝗮𝗶𝗻𝘀: Outperforms traditional LSTMs and competes favourably with Transformers in language modeling tasks. 🌐 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Suitable for building large-scale models, including Large Language Models (LLMs), thanks to favorable scaling properties. 🌍 𝗩𝗲𝗿𝘀𝗮𝘁𝗶𝗹𝗶𝘁𝘆: Applicable across various domains — NLP, time-series analysis, reinforcement learning, and more. 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀: ⏳ 𝗖𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆: mLSTM's matrix operations are resource-intensive. 🔧 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀: Stabilising exponential gates requires careful tuning (hence the introduction of stabiliser gates). 🔄 𝗦𝗲𝗾𝘂𝗲𝗻𝘁𝗶𝗮𝗹 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗶𝗻 𝘀𝗟𝗦𝗧𝗠: sLSTM blocks may limit parallelisation compared to Transformer architectures. 🌱 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗠𝗮𝘁𝘂𝗿𝗶𝘁𝘆: As a new architecture, xLSTM doesn't yet have widespread community support. 👉 Check out the paper for more details: https://lnkd.in/gHrFXN_n Thanks to Clarence Cheung and Emerie Luchen Li for organising the conference! What are your thoughts on xLSTM's potential impact?

46

5 Comments

Using your M4 iPad Pro as an external GPU/Neural Processor for your AI models ... a follow-up to the Apple MLX post about Apple being the value solution for training and inference from a few weeks ago ... With the release of Apple's M4, the listed specs for the included Neural engine increased TOPS (trillions of operations per second) by about 5% when using INT8 precision. But direct benchmarking tells a little bit of a different story because Key Value caching is faster on the M4 and that can improve model performance more than increased TOPS in the M4 chip's neural processor. The only way to target models to run on the Neural Engine is to use CoreML (coremltools) or reverse engineer CoreML models yourself from the coremltools source. It's an open source tool written by Apple to port a model from various toolkits like PyTorch, Tensorflow, SciKit, etc. To force a model to run on ANE (the Neural Engine), target the output to run on CPU+ANE (and not GPU). Unlike an Nvidia GPU, Apple doesn't document how to directly observe CoreML and Neural Engine performance metrics. When you build for an Apple handheld, you have to iteratively benchmark different approaches on different hardware to make choices about model, cache, stepwise iteration, and other algorithmic approaches. MLX, Apple's tool to allow multi-Mac neural networking inference and training through Thunderbolt chained systems, doesn't support the use of the Neural Processor. But a quick decomposition of MLX shows that it is possible to use the MLX approach for training/inference on multiple systems with a mixture of ANE, CPU, and GPU on each system. It's hard to use ANE because it is so opaque. But some algorithms will benefit from using ANE for interim inference steps. It is a good exercise for a student to expand and write about multi-machine usage of ANE within the MLX framework. So, for clarity, it is possible for students and other value seeking ML developers to use an iPad Pro as an external GPU and Neural Processor during algorithmic development by using a modified version of MLX. I may show an example, but I hope someone beats me to the punch with a more generalized version of MLX.

14

Ever wonder if the so-called long-context LLMs is really working ? Or just getting lost-in-the-middle ? This new benchmark, LongMemEval, tests the long-term memory of chat assistants. What's in it ? LongMemEval has: - 164 topics, & - 500 manually created questions and it tests 5 core memory abilities: - information extraction - cross-session reasoning - temporal reasoning - knowledge updates - abstention "Prior benchmarks often fail to capture the nature of user-AI interactions, only have limited memory ability coverage, or often feature too short chat history..." To address this, "LongMemEval requires chat systems to parse the dynamic interactions online for memorization, and answer the question after all the interaction sessions." They also discover several interesting things about how to deal with the memory in long-context LLMs, which may be of interest to those involved in such research But most importantly, they seem to verify what I suspect a lot of already know--truly interactive and useful long context LLMs are still a long-ways away. paper: https://lnkd.in/gTjTxAum code: https://lnkd.in/gC5CnQTq source: https://lnkd.in/g_C_iJDM

34

1 Comment

🚀 Exploring Redundancy in Transformers: Insights from "What Matters In Transformers?"🚀 The paper "What Matters In Transformers?" https://lnkd.in/dCyB4YHP presents a fascinating discovery: nearly half of the attention layers in large language models (LLMs) like Llama can be removed without significantly affecting their performance. This finding opens up new possibilities for optimizing transformer architectures. Key Takeaways: - The study experimented by deleting attention layers, MLP layers, or entire transformer blocks. While removing entire blocks or MLP layers led to notable performance drops, cutting attention layers caused almost no degradation. - In Llama 2 70B, removing half of the attention layers resulted in a 48% speed-up, with only a 2.4% decrease in benchmark scores. Recent results for Llama 3 show similar patterns. - The attention layers were identified for removal based on a cosine-based similarity score, highlighting redundancy where input and output were nearly identical. This process is a one-shot operation, without the need for retraining, although retraining could potentially recover some lost performance. These findings suggest opportunities to combine layer removal with other model compression techniques like pruning and quantization, leading to even greater efficiency. However, it’s important to note that the study focused on academic benchmarks (HellaSwag, MMLU), so the impact on conversational tasks remains to be explored. A simple yet powerful exploration of computational redundancy, offering valuable insights for anyone optimizing LLMs. Credit to Sebastian Raschka, PhD for highlighting this intriguing study. #AI #Transformers #MachineLearning #ModelOptimization #Llama #DeepLearning CC H.E. Abdullah Alswaha Lamya Abdulaziz AlOmair, PhD Latifa Al-Abdulkarim Nada Al Jeffri Deemah AlYahya Dr. Abeer S. Al-Humaimeedy Shahd AlSaba Nada Al Jeffri Farah AlRasheed Paolo Sironi Areej Al-Wabil Maha Arkoubi Nesreen Elofi Amr Almasri Najiba Jarraf Abdulrahman AlEissa Nisreen Al Khatib Ebrahim Baeshen Nicolas Babin Franco Ronconi Jason Dunn Tony Moroney د. ريم العطاس Nora S. Alturayeif, PhD Amna Hariri Zohour AlSaif , EMBA, CRMO Amal Merzouk Reema Alzahrani ملكة Maleka Alkebsi Meral AlHelwah Rand Alshehri رند الشهري Khaled Al-Bassam, خالد البسام Arwa Alnitaifi Yasser Aljarbou Fares AlOtaibi Reef AlThunayan Abdulaziz Al Meer

30

4 Comments

Can we trust LLM benchmarks(Like. MMLU, C-Eval)? A new study unveils the risk of data leakage and offers a solution to detect it! 🕵️‍♀️ The widespread use of multiple-choice benchmarks for LLMs has a hidden risk: data leakage. If a model has seen benchmark data during pre-training, it can simply memorize answers, leading to inflated performance scores. Here's the gist: ➡️ By shuffling the content of multiple-choice options, create derived datasets. ➡️ Analyze the log-probability distribution of an LLM's responses to these shuffled datasets. ➡️ A maximum log-probability accompanied by an outlier strongly indicates data leakage. Key findings: ⚠️ Leakage is a serious concern: The study evaluated 31 popular open-source LLMs, including LLaMA, Qwen, and Yi, on benchmarks like MMLU, CMMLU, C-Eval, and CMB. 🏆 Qwen family of LLMs exhibited the highest degree of data leakage across multiple benchmarks. 📉 Data leakage significantly inflates performance metrics, making it difficult to accurately assess LLM capabilities. Paper: https://lnkd.in/gfTxD3Ry Code: https://lnkd.in/gAeTaSbc #LLM #AI #DataLeakage #Benchmarking #MachineLearning #DeepLearning #AIethics #ResponsibleAI #NLP #LLaMA #Qwen #Yi

2

Claude vs ChatGPT: A 2024 Comparison As the AI assistant landscape evolves, Claude and ChatGPT remain at the forefront. Here's how these models compare in 2024: Capabilities: - Claude 3.5 showcases improved performance across various tasks compared to GPT-4. - Claude excels in reasoning, analysis, and following complex instructions. - ChatGPT shines in creative writing and casual conversation. Benchmark Performance: - On the MMLU benchmark:   Claude 3.5: 86.7%   GPT-4: 83.9% - On mathematical reasoning tasks:   Claude 3.5: 92% accuracy   GPT-4: 89% accuracy Limitations: - Neither model achieves perfect factual accuracy or real-time knowledge. - ChatGPT boasts broader integration with third-party apps and services. - Claude adheres to stricter ethical constraints in certain areas. User Experience: - Claude offers concise responses for simple queries. - ChatGPT tends to provide more detailed output. - Claude excels in breaking down complex tasks step-by-step. The competition between these AI assistants drives rapid advancements in the field. Users may benefit from trying both to determine their specific needs. That is the approach I took. For more insights on this comparison or to explore other facets of Claude vs ChatGPT, feel free to reach out for elaboration or additional details.

3

UDBRNet: A novel uncertainty driven boundary refined network for organ at risk segmentation https://buff.ly/4cngPxx

1

Since I had some spare time during the weekends I put something together: https://lnkd.in/eyjqrzbn This is the start of a new journey where I work on my passion of applying modern ML/NLP/AI/GNN techniques for Source Code (and Binary) Vunerability detection and Reverse Engineering. It features a new site, that I personally built, and where I planning to publish my personal Research and Tools. #NLP, #BERT, #CodeBERT, #GraphCodeBERT, #Transformers, #Cybersecurity

13

Here is a geeky post after a while (and long overdue)! This recent paper ( https://lnkd.in/eX7rKSvS ) about a new neural net as an alternative to the oldest NN, the MLP caught my attention: Kolmogorov-Arnold Networks (KANs) have learnable activation functions, compared to Multi-Layer Perceptrons (MLPs) with fixed activation functions. According to the paper, this architectural modification makes KANs significantly outperform MLPs in terms of accuracy, avoids catastrophic forgetting, and importantly makes them interpretable. The catch? Learning time is 10x slower and requires more computational resources (basically optimization in a higher-order space takes longer!) The KAN code is at https://lnkd.in/eA3-Nam4 . I might just try this out in my spare time!

38

3 Comments

Python was invented in the late 1980s by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABC programming language, which was inspired by SETL capable of exception handling and interfacing with the Amoeba operating system.Its implementation began in December 1989. Van Rossum shouldered sole responsibility for the project, as the lead developer, until 12 July 2018, when he announced his "permanent vacation" from his responsibilities as Python's "benevolent dictator for life" (BDFL), a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker(he's since come out of retirement and is self-titled "BDFL-emeritus"). In January 2019, active Python core developers elected a five-member Steering Council to lead the project. Python 2.0 was released on 16 October 2000, with many major new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support.Python 3.0 was released on 3 December 2008, with many of its major features backported to Python 2.6.xand 2.7.x. Releases of Python 3 include the 2to3 utility, which automates the translation of Python 2 code to Python 3. Python 2.7's end-of-life was initially set for 2015, then postponed to 2020 out of concern that a large body of existing code could not easily be forward-ported to Python 3. No further security patches or other improvements will be released for it. Currently only 3.8 and later are supported (2023 security issues were fixed in e.g. 3.7.17, the final 3.7.x release. While Python 2.7 and older is officially unsupported, a different unofficial Python implementation, PyPy, continues to support Python 2, i.e. "2.7.18+" (plus 3.9 and 3.10), with the plus meaning (at least some) "backported security updates".#snsinstitutions #snsdesignthinkers #designthinking

1

That's awesome! Mistral's new features for building custom LLMs sound incredibly developer-friendly. The ability to fine-tune and create custom agents makes it a powerful tool for personalized AI solutions. Thanks for sharing the video link—can't wait to check it out! 🚀🔍 #CustomLLMs #AIInnovation #DeveloperTools #MachineLearning #MistralLLM

1