Awadelrahman Ahmed

RAG, or Retrieval-Augmented Generation, is a machine learning technique that combines information retrieval with language generation. In RAG, a language model (often a large language model, LLM) is used in tandem with a retrieval model to generate accurate and relevant responses based on both the model’s trained knowledge and an external set of documents or data.

6

📚 Another conference, another article! The International Conference on Machine Learning (ICML) took place last week, from July 21 to 27, 2024, in Vienna. In this article I explore some exciting advances in Machine Learning presented at this year’s edition: 🔍 For knowledge transfer, I highlight LLM distillation, 2-bit quantization, and transfer learning, which aim to create smaller and efficient models. 📈 In time series analysis, I explore zero-shot forecasting and metadata-enhanced time series generation, which is especially useful in energy field. Discover more in my medium article!

21

We used to focus on performance when we were writing code. It won’t be difficult for us to get thousands of articles if we search “python performance”. However, I rarely see articles that tell us to optimise the memory consumption of our code. Of course, performance is critical in programming, but memory consumption will decide how much money we need to spend on hardware. Consider that we are going to train a model on a big dataset. We may need a large VM with 64GB RAM to do the job. However, if we can optimise our code, 32GB RAM may be able to achieve a similar elapsed time. In that case, we may save a lot of money on hardware resources. Before I can optimise our code to reduce memory consumption, we need some effective ways to measure it. Also, by knowing the approaches to measuring memory consumption, we will be able to quantify and benchmark it. So, in this article, the common memory-measuring approaches that I use to leverage will be introduced. #python #python3 #datascience #datascientist #dataengineer #dataengineering #datapreprocessing #machinelearning #ai #artificialintelligence #dataanalytics #dataanalysis #datamining #softwareengineering #softwaredevelopment #softwaredesign #programming #pythonprogramming https://lnkd.in/gB45YUxs

16

A Comparison of Top Embedding Libraries for Generative AI https://lnkd.in/dGiBEB5U

3

Sharing my experience of making use of low-code, open-source models in #KNIME to evaluate time series forecasting for pricing estimates as part of their weekly challenges. My key takeaways are as follows -- 📌Even though new models like Prophet and Neural Prophet are more sophisticated and simpler to use, traditional models like SARIMA and ARIMA can still perform better than these modern models under given circumstances. 📌Using pre-built KNIME (low-code no-code) environments can save a lot of time for business experts, allowing them to focus on #business #insights rather than backend development activities. 📌To uncover a viable solution, it is essential to explore different models and fine-tune parameters tailored to the unique business challenge at hand, rather than jumping to quick conclusions. 📢In today's dynamic landscape, a one-size-fits-all approach simply will not cut it. Collaborating with Subject Matter Experts, including Data Scientists and Statisticians, is essential to elevate initial prototypes into advanced working models. These models empower businesses to make informed decisions in real-time, driving success and innovation. Look forward to hearing from like-minded peers in the comments section. #KNIME #timeseries #businessinsights

10

This book is a must-read for anyone working with data.    https://lnkd.in/dZMa5yrZ   I read it a few years ago, and every so often, I revisit certain chapters or even the entire book, especially when I need to explain complex topics in simple terms. The book dives into fundamental statistical concepts like correlation, causation, probability, and regression, using real-world examples to demonstrate their significance and practical applications.    I’d highlight these key topics:    What causes what? For instance, do statins actually reduce heart attacks and strokes?  How confident can we be in our findings? This section covers estimates and intervals, emphasizing the importance of uncertainty intervals (not just point estimates) in statistical accuracy.  Learning from experience: The Bayesian approach. #statistics #foundations #Data  

8

2 kommentarer

Generative AI and Python

1

How valuable is a customer to your company in the long run? 🤔 Dive into the concept of Customer Lifetime Value (CLTV) with my latest article and video series on DevGenius ! Learn about the colorfully named "Buy Till You Die" modeling technique and how to leverage an open source Python library, PyMC Marketing that uses probabilistic programming 🎲 to extract insights from customer transaction data to improve customer relationships and reduce churn. 📈 https://lnkd.in/gCN4EdHK PyMC Labs Community Manager PyMC #CustomerLifetimeValue #CLTV #MarketingAnalytics #CustomerRelationships

5

6 kommentarer

Here is another misuse of Artificial Intelligence If the science and physics of the state of the elements (solid, liquid, vapor, and plasma) is already settled and understood, why people insist in using machine learning to rediscover what we alll already know? To me it is a symptom that pseudo-AI companies are pushing researchers to use their software to recreate physics with only data. I have seen this happening in respected oil and gas companies. Usually this occurs when the IT departments take too much control of the data science and machine learning workflows that engineers use every day. My message is this: do not succumb or fall in the temptation of so-called AI shiny software. Know and learn you science and physics first. Keep in mind that marketing or sales people are not in their majority engineers or scientists. #artificialIntelligence #AI #spe #petroleumEngineering Massachusetts Institute of Technology

40

5 kommentarer

We’re likely familiar with the tendency of Large language Models (LLMs) to hallucinate which underscores the importance of maintaining critical perspective when engaging with them. A key factor in assessing the reliability of these models is their capacity for logical reasoning. This prompted me to explore the question of how proficient are LLMs at tackling causality problems. Recently I’ve been reading some literatures on LLMs and causality and I find this recent paper by Microsoft Research quite interesting. They found that LLMs like GPT-3.5 and GPT-4 have shown significant capabilities in addressing causality problems, even surpassing existing state-of-the-art algorithms! They can generate causal graphs through leveraging metadata and domain knowledge, achieving high accuracy in tasks like pairwise causal discovery and full graph generation. Counterfactual reasoning is also something that they excel at, a crucial aspect of causal analysis that involves exploring hypothetical “what-if?” scenarios to understand the potential outcomes of alternative events or decisions. As advancements continue in this field, it's really exciting to consider how these capabilities might reshape our interaction with AI in the future. #LLM #deeplearning #causality #AI #generativemodel link to the paper: https://lnkd.in/eqxQ63am.

10

📍Kolmogorov-Arnold Networks - Chapter 1 “100x more accurate and parameter efficient than MLPs at modelling ODEs” - Math and how it compares to a Multi Layer Perceptron - Architecture and KAT (approximation theorem) - Challenges In the next post(s) - applications, use cases and limitations

44

3 kommentarer

As one of the most unique features in Python, decorator might not be quite easy to understand. However, it will provide super conveniences to our development no matter if you are a Data Scientist, Data Engineer or Web Developer. A few years ago, I wrote a tutorial article about Python decorator. That is highly recommended if you are after some basics about the Python decorator (link below). https://lnkd.in/gF3PJXG5 In this article, I will summarise five different scenarios of Python decorator usage. So, hopefully, it will give you ideas about when and why we should use decorators. #python #python3 #programming #pythonprogramming #datascience #artificialintelligence #ai #datascientist #dataengineering #dataengineer #dataanalytics #dataanalysis #datamining #applicationdevelopment #softwaredevelopment #softwareengineering #softwaredesign https://lnkd.in/gmJUkGss

25

A few weeks ago, I wrote another article to explain the mechanisms and logic behind some popular “Python Tricks”. One of them is to use Python Set but not List when we can. After this article became popular, many readers asked me or argued that Python Set isn’t always fast. That’s absolutely true. Therefore, I decided to write this article to have a deep dive into the data structures of Python List and Set. In this article, I’ll first use practical code to compare the performance between Python List and Set in different scenarios. Then, I’ll introduce the data structures they used, which are Dynamic Array and Hash Table. Based on the characteristics of these data structures, I’ll explain why Python List or Set has better performance in certain scenarios. #python #python3 #datascience #datascientist #dataengineer #dataengineering #datapreprocessing #machinelearning #ai #artificialintelligence #dataanalytics #dataanalysis #datamining #softwareengineering #softwaredevelopment #softwaredesign #programming #pythonprogramming https://lnkd.in/gaYqBjhk

7

🚀 Dive into "Introduction to LLMs: A Component Overview" in our Data Mastery Series! Explore the core components of Large Language Models (LLMs), including ChatOpenAI system messaging, differences between LLM models, and essential LLM parameters like temperature and top_p. This episode is perfect for anyone looking to grasp the functional depth of LLMs in practical applications. Join us for a detailed exploration! #DataScience #MachineLearning #Learning

1

So everyone agrees AI is powerful, but why is implementing lasting AI tools in enterprises so difficult? How can we effectively get this technology to  employees? And what are the really valuable use cases to focus on? These questions occupied our CoC Data Science team @Wintershall Dea over the past few years. I have always loved my work, and as I prepare for a break, I want to share my top 10 learnings: 1. You can’t do Data Science alone. You need a team. 2. You can’t do Data Science without data foundations, i.e. Data Governance and integration. You need skilled data managers and data engineers to support your data scientists. 3. You need MLOPs (Machine Learning Operations) and your MLOPs skills are equally if not more important than your Data Science skills to operationalize (i.e. automate pipelines, staging, deployment etc). Once you reach 10 use cases, an MLOPs strategy needs to be in place. 4. Think about tool maintenance early on, not just when you reach MVP.  Consider who will maintain the tool and the associated costs. Data Scientists love solving complex problems, but typically won't stay on the same project for years. The maturity of digitalization teams can be gauged by how well they operationalize and maintain use cases. 5. Know your tech stack (Python, Azure, or another). Get certified. Understand the economics (FinOPS) 6. Implementing Data Science in a non-tech company requires co-development with third parties. However, in-house skills are necessary for implementation, integration, knowledge retention, reducing dependency, and understanding value and effort. Building these capabilities takes time, commitment and trust. 7. If you do in-house (co-) development, write your tests and document. :) 8. Enabling colleagues and building up an innovative company culture will help you find disruptive use cases and retain talent. However, it requires significant energy and effort and is challenging to economically quantify and to 'sell' to management. 9. LinkedIn is the most effective channel for reaching management and (self-)branding 10. The most hardworking and kindest colleagues often remain behind the scenes. Shout out to Prihandono Aditama for tirelessly giving his all for the team. I can't take this break without thanking my colleagues, with whom I've had an incredible journey. There are simply too many to tag. I have met incredibly smart and passionate professionals. My IT, D&T colleagues were fantastic; the support from #IBM was amazing. And not to forget the real heroes: the colleagues in the business units. Before going on my break, I had the pleasure of attending the #EAGEAnnual this week. Let me highlight two individuals I had the pleasure to meet, who work tirelessly to share knowledge and connect AI professionals in the industry. Thank you for hosting a great LLM deep dive workshop, George Ghon and Lukas Mosser. Also, thanks to Nicole Grobys for her efforts in sharing ML applications in the subsurface domain.

76

17 kommentarer

From Novice to Expert : Building a Strong Data Science Profile. Read my article if you want to start your journey in Data Science.

28

1 kommentar

🚀 Demystifying Multiclass Classification: OvA vs. OvO Strategies! 🤖 When tackling real-world classification problems, we often move beyond simple binary outcomes (e.g., spam vs. not spam) to multiclass classification tasks, where there are multiple possible labels (e.g., handwritten digits 0-9). But did you know that there are specific strategies to adapt binary classifiers for these tasks? Enter One-versus-All (OvA) and One-versus-One (OvO)! 🔹 One-versus-All (OvA) This approach trains one binary classifier per class. For example, if you're classifying images of digits, you'd have 10 classifiers: one for each digit (0 vs. all, 1 vs. all, etc.). The input is passed through each classifier, and the class with the highest decision score is chosen. Pros: Simpler setup with fewer models. Best For: Algorithms that scale well with the size of the training set. 🔹 One-versus-One (OvO) In OvO, a binary classifier is trained for each pair of classes, resulting in N×(N−1)/2N \times (N - 1) / 2N×(N−1)/2 classifiers (e.g., for 10 classes, that's 45 models!). During prediction, the input is evaluated by all classifiers, and the class with the most "wins" is selected. Pros: Each model is trained on smaller subsets, which is ideal for algorithms like SVMs that don’t scale well with large datasets. Best For: When training many smaller models is preferable to training fewer, larger ones. Understanding these strategies not only helps optimize your machine learning workflow but also deepens your appreciation for how algorithms scale and perform across various tasks. Next time you face a multiclass problem, remember that your choice between OvA and OvO can make all the difference! 💡 Which strategy do you prefer for multiclass problems? Share your thoughts below! 👇 #MachineLearning #DataScience #MulticlassClassification #OvA #OvO #AI

1

Meet 𝗠𝗼𝘀𝗵𝗶 from Kyutai 👋 The first ever 𝗦𝗽𝗲𝗲𝗰𝗵 → 𝗦𝗽𝗲𝗲𝗰𝗵 + 𝗧𝗲𝘅𝘁 Large Language Model🤖 Magical progress made by a 𝘀𝗺𝗮𝗹𝗹 yet 𝗳𝗼𝗰𝘂𝘀𝗲𝗱 team in 𝟲 𝗺𝗼𝗻𝘁𝗵𝘀 The best part is it is 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲 Link to speak with 𝗠𝗼𝘀𝗵𝗶 is in the pinned comment 📌 𝗧𝗿𝘆 𝗶𝘁 and 𝘀𝗵𝗮𝗿𝗲 𝘆𝗼𝘂𝗿 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 as a 𝗰𝗼𝗺𝗺𝗲𝗻𝘁 Spread the word by Reposting🔄 https://lnkd.in/g-_FrQrX

5

2 kommentarer

Can the latest GPT model, o1, truly tackle PhD-level challenges? Some might say yes—but as shown in the attached example, even this advanced AI stumbles on some of the most debated scenarios. What’s fascinating is that this very sentence has likely sparked countless conversations, influencing fine-tuning data. Yet, errors persist. What does this mean for organizations aiming to harness Generative AI? 🤔 Key Takeaways for Corporates & Institutions: 1️⃣ Human in the Loop: Critical for validation and oversight. AI is smart, but not infallible. 2️⃣ Subject Matter Experts: Still irreplaceable in ensuring accuracy and domain-specific insights. 3️⃣ Continuous Evaluation: Unlike traditional deterministic algorithms, GenAI evolves, requiring ongoing checks. 4️⃣ True Tech Expertise: To maintain reliable, production-grade outputs, expert guidance is non-negotiable—you can’t afford AI making elementary mistakes in front of clients or stakeholders. AI is powerful, but it’s the synergy with human expertise that unlocks its true potential. 🚀

4