Mehmet Kutluay, PhD

When invited to the Super Data Science Podcast, Ritchie Vink, author of Polars, mentioned an upcoming Polars book set to launch in January 2025: "Python Polars: The Definitive Guide" by Jeroen Janssens and Thijs Nieuwdorp. The official Github repository already has a lot of code examples that worth checking out: https://lnkd.in/dJGbwQnM The Early Release version can also be read on O'Reilly Learning platform: https://lnkd.in/daTEf8Tv The book is offering guidance for you to:  - Process larger-than-memory datasets at record speed - Apply the eager, lazy, and streaming APIs of Polars and decide when to use which - Transition smoothly from Pandas or Spark to Polars - Integrate Polars into your existing codebase - Work with Arrow and Parquet to efficiently read and write data - Translate complex ETL tasks into efficient and elegant queries I started reading the Early Release version and I am enjoying it so far. I am looking forward to the final version of the book. #Python #DataFrameLibrary #Polars

40

2 commentaren

🚀 Exciting developments in Large Language Models! Our latest blog post, written by Jeroen Overschie, discusses Retrieval Augmented Generation (RAG), a technique that transforms data access and reduces hallucinations. 🗝️ Key Insights: Level 1️⃣ - Basic RAG - Fundamentals of document retrieval and answer generation. Level 2️⃣ - Hybrid Search - Combining vector and keyword search for better accuracy. Level 3️⃣ - Advanced Data Formats - Handling complex formats like HTML, Word, and PDF. Level 4️⃣ - Multimodal Models - Using models that understand text, audio, and images. RAG is rapidly evolving in the industry. Whether you're a seasoned professional or just embarking on your journey, this post provides actionable steps for implementing RAG. ⛓️ Read more here: https://hubs.ly/Q02LptM40. #ai #machinelearning #rag #xebia

68

3 commentaren

🎤 "Many open-source projects are strongly reliant on the goodwill and availability of one person or a very small group of people." Read Sebastiaan Mathot's journey in developing 𝗢𝗽𝗲𝗻𝗦𝗲𝘀𝗮𝗺𝗲, a research software tool for building behavioral experiments 👉🏻https://lnkd.in/eRgjap7U #opensource #researchsoftware #sustainability

10

1 commentaar

In this Technical Deep Dive, we share our approaches in using Large Language Models (#LLMs) to make numerical-type data(sets) like DHS, JMP, HDX more accessible with the example of WASH data. In this 15 min. video Olivier Mills and team cover: 1️⃣ Basic scripting to extract core data from datasets from API endpoints 2️⃣ How to use LLMs to generate LLM-friendly schema definitions 3️⃣ How to structure a AI-friendly database store that can scale to thousands of datasets and millions of data indicators 4️⃣ How to use vector querying to retrieve relevant indicators from the store based off of a natural language (english) query 5️⃣ How to build a prompt that generates valid SQL query using LLMs with the above retrieved data schema. The various components form a single AI system combining data ingestion and data exploration using a software layer, a data layer and multiple LLM layers. ⏭️ Next steps will be to expand the datasets, automate other data source ingestion and further test the system with more complex plain english queries. We plan to incorporate this as a new feature for WASH AI after testing. Feel free to reach out if you would like a copy of the Notebooks, sample prompts and data schema definitions. We are #opensource The initial work was done in collaboration with Data for Good YYC (see https://lnkd.in/g3AijS93)

3

1 commentaar

in Pandas 2.2 one can set pyarrow as the default backend. in this presentation the advantages of arrow vs numpy are presented: 1. Faster string operations 2. Consistent missing value handling 3. Efficient data serialization Minute 24 https://lnkd.in/gbbGpR56

22

⭐ Unlocking Math Performance Insights with Educational Data Mining! ⭐ Congrats are in order for SEERNet researcher Avery Closser and her co-authors! Their new publication, "Exploring The Impact of Symbol Spacing and Problem Sequencing on Arithmetic Performance: An Educational Data Mining Approach" is out now in the Journal of Educational Data Mining. Their research reveals how visual features and problem order impact student math performance. Key findings include faster problem-solving when higher-order operators are on the left side and varied response times based on problem sequence. These results highlight the importance of considering problem order and diverse analysis methods in online math experiments. The study has implications for researchers using log data from online settings to improve educational outcomes. https://bit.ly/462jTNI #EdResearch #MathEducation #EdTech #LearningScience #EducationalDataMining

3

1 commentaar

How do you measure the effectiveness of your text generation models?  One metric you can consider is 𝗥𝗢𝗨𝗚𝗘-𝗟. ROUGE stands for 𝗥𝗲𝗰𝗮𝗹𝗹-𝗢𝗿𝗶𝗲𝗻𝘁𝗲𝗱 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝘂𝗱𝘆 𝗳𝗼𝗿 𝗚𝗶𝘀𝘁𝗶𝗻𝗴 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 🫡 It’s commonly used for assessing → summarisation → translation → question answering → pairs of reference / prediction The method measures similarity > between a 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗲𝗱 text (e.g., from an LLM) and > a 𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 text. It relies on the 𝗟𝗼𝗻𝗴𝗲𝘀𝘁 𝗖𝗼𝗺𝗺𝗼𝗻 𝗦𝘂𝗯𝘀𝗲𝗾𝘂𝗲𝗻𝗰𝗲 (𝗟𝗖𝗦), which identifies the longest sequence present in both texts. It comes in different flavours > 𝗣𝗿𝗲𝗰𝗶𝘀𝗶𝗼𝗻: Proportion of LCS in the predicted text relative to its total length.   > 𝗥𝗲𝗰𝗮𝗹𝗹: Proportion of LCS in the reference text relative to its total length.   > 𝗥𝗢𝗨𝗚𝗘-𝗟 (𝗙-𝗺𝗲𝗮𝘀𝘂𝗿𝗲): The harmonic mean of Precision and Recall, giving you a balanced score that you may find most useful. 𝗣𝗿𝗼𝘀 > word order captured > efficient, not pre-trained model needed > good for sentence-level overap > good for extractive tasks 𝗖𝗼𝗻𝘀 > fail at semantic similarity and synonyms > tends to favor longer outputs > struggles with creative tasks I think it is quite light so it probably doesn't hurt using it. → You don't need to just look at one unique metric. It is OK to use severals 🙂 Here is a notebook with example LLM evaluations implemented. https://lnkd.in/e5w6Ewbv --- Follow for more tips and guides on LLM usage, LLM datasets and LLM evals.

26

2 commentaren

🎬 New Research Video Alert! 📽️ How can open data intermediaries address challenges in the open data ecosystem? At the recent #DataforPolicy2024 conference, Ashraf Shaharudin, Bastiaan van Loenen, and Marijn Janssen from Delft University of Technology have delved into this important question with their paper, "Envisioning Contributions and Types of Open Data Intermediaries" In this comprehensive research, the authors explore how open data intermediaries—third-party actors providing specialized resources—can tackle challenges in the open data ecosystem (ODE). Their study, which includes interviews and validation exercises in the EU context, identifies 20 potential contributions of these intermediaries and addresses 27 challenges related to supply, flow, use of open data. Watch the full presentation on YouTube: https://lnkd.in/eP8E4SQd Discover the vital role these intermediaries play in enhancing open data sustainability and the incentives needed to maximize their impact. To watch the playlist of all Data for Policy 2024 full-length presentations: https://lnkd.in/eWUagiT8 #OpenData #OpenDataIntermediaries #DataForPolicy2024 #Innovation #Transparency #Research #TUDelft

4

An interesting post on how Skyscanner uses the combination of Dr Jekyll and its in-house statistical tool called WISE to ship experiment variants to production. Instead of using a frequentist approach as is done in most AB experiments, this framework uses a Bayesian approach to quantify the uncertainty in the final estimates by reporting credible intervals instead of confidence intervals. Summary: - Example of an experiment to test whether a pop-up, mentioning that ‘Skyscanner never takes a cut’ increases the ‘redirector rate’ or not. - If sufficient data has not yet been collected, WISE’s results will say ‘keep testing’. - WISE is able to provide a recommendation on the variant to ship. - WISE uses the Bayesian approach. They model their belief of the underlying parameter values as probability distributions, which are updated as new data is observed. - For conversion metrics, it uses a beta-binomial model - For revenue per user metrics, it uses a hurdle gamma-exponential model - What's the experiment stopping criteria: Through Monte Carlo simulations of thousands of experiments, they have determined a set of bespoke target 90% HDI widths to serve as their stopping criterion, which ensured an appropriate balance between the duration of experiments and the correct decision rate. https://lnkd.in/dv7BEHXe #statistics #analytics #rstats #python #peopleanalytics #experimentation

6

1 commentaar

🚀 An Overview on Retriever Models: Optimizing query-to-context matching 🛡️ In our upcoming blog, we dive into the pivotal role of Retriever models within RAG pipelines. These models are essential for extracting the most relevant documents or articles in response to a user's query, effectively bridging the knowledge gap that exists in large language models with a knowledge cut-off date. By integrating these Retriever models, LLMs can provide more accurate, timely, and contextually relevant responses, significantly enhancing the overall user experience in applications that rely on up-to-date information. Checkout the full blog here: https://lnkd.in/dh8Pj2wW #AI #ArtificialIntelligence #LLM #ConversationalAI

9

Great Tables is now BYODF (Bring Your Own DataFrame). The Great Tables team have supported both Polars & Pandas DataFrames for a while, and have finally removed our hard dependency on Pandas. This means that you can still use either DataFrame, but don't have to install the other in order to use Great Tables. Removing dependencies is a little thing that goes a long ways for helping users in restricted computing environments, or who want a lightweight install. In this post, Michael Chow goes into detail on: * 👯‍♀️ The things great_tables currently depend on. * 🏗️ The challenge of removing hard dependencies. * 🫠 The approach chosen for making Pandas optional. https://lnkd.in/gsuB7dhj

88

🚀 uv (Python) enabling new workflows! Last night I had an idea: what if we could open any Jupyter notebook from GitHub with all its dependencies installed instantly? The result 2 hours later: opennb (https://lnkd.in/gGZ6CDPN) Example: uvx --with "pipefunc[docs]" opennb pipefunc/pipefunc/example.ipynb This single command: ✨ Creates a temporary environment 📦 Installs dependencies (instant with uv's cache!) 📓 Downloads and opens the notebook What made this possible? The amazing uv tool (https://docs.astral.sh/uv/), which handles Python package management at insane speeds. The best ideas are often the simplest - combining uv's speed with Jupyter's accessibility to make trying out Python tutorials friction-free. #Python #JupyterNotebook #OpenSource #DeveloperTools #Programming

92

5 commentaren

I've just published a new article about my thoughts on Rust in data science programming. In this post, I explore: - How Rust can supercharge your data processing pipelines - The benefits of Rust's safety features for robust code - Ways to integrate Rust with thx existing Python workflows Whether you're a Python enthusiast curious about Rust, or a data scientist looking to level up skills, this article offers insights into how Rust can complement your data science toolkit. Check out the link below.

3

"In this article, we first explore splines, as they help us understand the architecture and key elements of KAN. Then, we make a deep dive inside the inner workings of KAN. Finally, we apply KAN to time series forecasting, and evaluate its performance against the standard MLP and the N-BEATS model." by Marco Peixeiro

25

"In this article, we first explore splines, as they help us understand the architecture and key elements of KAN. Then, we make a deep dive inside the inner workings of KAN. Finally, we apply KAN to time series forecasting, and evaluate its performance against the standard MLP and the N-BEATS model." Marco Peixeiro offers a hands-on introduction to Kolmogorov-Arnold Networks.

71

📘 Just published a new blog post diving into the math behind self-attention in #DeepLearning! 🤓 I'm a visual learner - until I get a visual intuition behind a concept it's very hard for me to understand it properly. This blog is such an attempt to understand self-attention and how it is so effective. Thanks to Luis Serrano for publishing an excellent video on this topic. A big thanks to Sebastian Raschka, PhD for this amazing book-Build a large language model from scratch. 🔗 https://lnkd.in/gUnWPdJ6 #nlp #attention #MachineLearning #Datascience

365

2 commentaren

Has anyone tried Claude from Anthropic? I find its depth in processing and structuring really awesome! There are other chatbots beside ChatGPT, like Grok, but it is not yet available in Europe. https://lnkd.in/edkStR5J #chatbots #artificialIntelligence #intuition #analysis

4

🚀 An Overview of Retriever Models: Optimizing Query-to-Context Matching 🚀 In our upcoming Miko AI blog, we will explore the pivotal role of Retriever models within RAG (Retrieval-Augmented Generation) pipelines. These models are essential for extracting the most relevant documents or articles in response to a user’s query. - They effectively bridge the knowledge gap in large language models caused by knowledge cut-off dates and gaps. - They also enable us to ground the LLM by ensuring that responses are rooted in factual, sourced data, helping to counter bias, reduce hallucinations, and mitigate inaccuracies. - LLMs can provide more accurate, culturally appropriate, and contextually relevant responses, significantly enhancing the overall user experience This makes them invaluable for improving the reliability and trustworthiness of the generated content. Check out the full blog here: https://lnkd.in/dh8Pj2wW #AI #ArtificialIntelligence #DeepLearning #LLM #RAG #ConversationalAI

9