Ram Parikh

Something I've written on how a T20 batting innings can be assessed. This framework can be extended to assessment of individual players as well, which I will do in the susequent posts. For now, please feel free to comment and if you like the content, please subscribe. #cricket #cricketanalytics #data #dashboards #datavisualization #sportsanalytics https://lnkd.in/gb7T-UzJ

28

6 Comments

Sharing update from SSRN publishing... "Dear Ashutosh Bhatnagar: Your paper, "NEIGHBORHOODS OF PRIMES", was recently listed on SSRN's Top Ten download list for: Discrete Mathematics & Combinatorics eJournal and Number Theory eJournal. As of 02 October 2024, your paper has been downloaded 16 times. You may view the abstract and download statistics at: https://lnkd.in/eMmhurr4. Top Ten Lists are updated on a daily basis. Click the following link(s) to view the Top Ten list for: Discrete Mathematics & Combinatorics eJournal Top Ten and Number Theory eJournal Top Ten. Click the following link(s) to view all the papers in: Discrete Mathematics & Combinatorics eJournal All Papers and Number Theory eJournal All Papers. To view SSRN's Top Ten lists for any network, subnetwork, eJournal or topic on the Browse list (reachable through the following link: https://lnkd.in/eEmS8twm), click the "i" button to the right of the name, and then select the "Top Downloaded" link in the popup window. Your paper may be included in future Top Ten lists for other networks or eJournals. If so, you will receive additional notices at that time. Questions? Visit the SSRN Support Center. This email helps you gauge the impact of your paper. To unsubscribe from ALL notifications, including SSRN eJournals, click here. Thank you, The SSRN Team"

10

3 Comments

📢 Best Advice from a Senior Data Scientist Packed in One Podcast: Insights from Abhishek Das!...🌟 🔷 Speaker: Abhishek Das Senior Data Scientist at Aptus Data Labs Ex-HSBC 9+ Years of Experience in Advanced Analytics and Machine Learning 🔷 Hosted by: Avinash Dara Here are some highlights from his talk: 🔷 His Personal Life: • He came from an Electronics and Communication Engineering background with an interest in research. • He was always interested in the business field to scale. • In his college placements, he joined e2open where the environment was new to him. He took some time to understand and scale in that industry. After that, his data science career started. In addition to this, he also started writing a book. 🔷 He worked across diverse industries such as: • Transportation, pharmaceuticals, manufacturing, supply chain, e-commerce, and banking. • Each sector has shown him the immense value data science brings to businesses, driving growth through informed decision-making. 🔷 He explained what Data Science is: • Data science is leveraging data to add value to a business. • He gave a practical example like Uber and Ola, which calculate the ride price based on other factors. 🔷 Career Paths in Data Science: 1. Data Engineer o Roles: Building and maintaining data infrastructure o Skills to acquire: Data engineering tools and technologies 2. Data Analyst o Roles: Extracting data, proficient in SQL, visualization, and providing insights o Skills to acquire: Strong SQL skills, data extraction abilities, and expertise in visualization tools 3. Data Scientist o Roles: Focusing on machine learning, algorithms, optimization, and prediction o Skills to acquire: Proficiency in machine learning, algorithms, and predictive analytics 4. ML Engineer o Roles: Researching, creating, and improving algorithms, and deploying models o Skills to acquire: Deep understanding of algorithm development and deployment 🔷 His advice for interview tips: 1. Be truthful about your skills. 2. Clearly indicate your proficiency levels. 3. Showcase 2-3 relevant projects, including ML projects. 4. Align your projects with industry-relevant problems, such as financial predictions or sales forecasting. 🔷 His approach for finding opportunities: • Apply to startups and MNCs. • Optimize your resume on LinkedIn. • Network with hiring managers and senior employees for referrals. • Target analytics-driven companies. 🔷 He also explained about LinkedIn: • LinkedIn is an incredible platform for networking and professional development. • Maintain a problem-solving mindset and continually showcase your skills and projects. #datascience #dataengineering #mlengineer #machinelearning #ccbp #advise #podcaste #datascientist #learning

35

In 2014, India's average data consumption per user was just 0.3 GB/month at ₹269/GB. Today, thanks to Modi Govt's push and players like Jio, users consume 30.3 GB/month at ₹10-13/GB. Remember when we rationed data usage to stretch it for a month? Now, everyone's a legal expert, taxation guru, election analyst, military strategist and more! 😆 We've come a long way from just worrying about onion and tomato prices to debating complex tax issues like Indexation. Cheers to progress and thanks to Modi Govt for making us a more developed nation!

683

57 Comments

Question : Sampling Techniques : What are they❓ As we saw recently in exit polls on Indian elections, choosing samples from a population can have a significant impact on predictions. #AI enthusiasts need to have a solid understanding of statistical sampling techniques. Statistical sampling is essential for AI product managers because it helps them: 1️⃣ Understand the Target Population: Accurate sampling ensures that the data used to train and evaluate AI models is representative of the target population, reducing the risk of biased or skewed results. 2️⃣ Optimize Data Collection: Sampling techniques can help AI product managers determine the appropriate sample size and sampling method, ensuring efficient and cost-effective data collection. 3️⃣ Validate Model Performance: Sampling is crucial for testing and validating AI models, as it allows for the creation of representative test sets that accurately reflect the real-world scenarios the model will encounter. AI enthusiasts should be aware of the different types of sampling methods, including: 🔺 Probabilistic Sampling Techniques: methods where randomisation is used instead of deliberate choice. 👉 Simple Random Sampling: Each member of the population has an equal chance of being selected. 👉Stratified Sampling: The population is divided into subgroups (strata), and samples are drawn from each stratum. 👉Cluster Sampling: The population is divided into clusters, and a random sample of clusters is selected. 👉Systematic Sampling: Every nth member of the population is selected, starting from a random point. 🔺 Non-Probabilistic Sampling Techniques: methods where some selection criteria exists 👉Convenience Sampling: Selecting participants based on their availability and accessibility. 👉Quota Sampling: Selecting participants to match predetermined proportions in the population. 👉Purposive Sampling: Selecting participants based on specific characteristics or criteria. 👉Snowball Sampling: Selecting participants through referrals from initial participants. The choice of sampling method depends on the specific goals of the AI project, the available resources, and the characteristics of the target population. Choosing the wrong sampling technique can lead to biased results, skewed data, and ultimately, the development of AI models that fail to perform well in real-world scenarios. 💡 Quick Tip: When selecting a sampling technique, consider the diversity and representativeness of the target population, the level of precision required, and the resources available for data collection.  #AI #ArtificialIntelligence #productmanagement #productmanager #DataScience 

7

🌟 How to get started with GenerativeAi (RoadMap for Beginners) 🌟 💡Here’s a 5 Week RoadMap for anyone who would like to start their Generative AI Journey 1️⃣ Week 1: LLM Basics and Foundation Introduction to Large Language Models, covering their architecture, training techniques, and real-world applications. 📕Course: - Transformer course by HuggingFace - Introduction to GenerativeAi by DataCamp 2️⃣ Week 1-2: Prompt Engineering Exploration of Large Language Models, covering core concepts and advanced techniques for effective prompt design and optimization 📕Course: - ChatGPT PromptEngineering for Developers by DeepLearning.ai - Prompt Engineering with Llama2 by DeepLearning.ai 3️⃣ Week 2-3: Building LLM Application Developing powerful AI applications using Large Language Models and the versatile LangChain framework. 📕Course: - LangChain chat with your data by DeepLearning.ai - LangChain for LLM Application Development by Deeplearning.ai - Building Applications with Vector Databases by DeepLearningAI 4️⃣ Week 3-4: RAG and FineTuning Master the art of enhancing LLM performance through Retrieval-Augmented Generation and custom model fine-tuning for specialized tasks. 📕Course: - Augment your LLM using RAG by Nvidia - Knowledge Graphs for RAG by DeepLearning.ai - FineTuning LLM by DeepLearning.ai 5️⃣ Week 4-5: Evaluation & Implementation Putting your LLM model into production 📕Course: - LLM Ops by Deeplearning.ai - Building and Evaluating AdvancedRAG Applications by DeepLearning.ai (Link to courses in 1st comment) 🤔 The best way to update is to create real-life applications So here’s a Github repository containing GenerativeAI project :- (Link in 2nd comment) #GenAI #GenerativeAi #llm #chatgpt #learngenerativeai #generativeairoadmap #generativeaiguide #learngenai #ChatGPT  #chatbots #chatgpt4 #chatgpt3 #GenerativeAI  #ArtificialIntelligence #AI #MachineLearning #ml #AIroadmap #DataScience

31

4 Comments

Checkout these interesting insights on 𝗣𝘆𝘁𝗼𝗿𝗰𝗵 𝟮.𝟱 Release: 🔥 𝗞𝗲𝘆 𝗵𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: - 𝟳𝟱% 𝗦𝗽𝗲𝗲𝗱𝘂𝗽 𝗳𝗼𝗿 𝗟𝗟𝗠𝘀: New CuDNN backend for SDPA delivers up to 75% faster transformer execution on H100 GPUs. - 𝗥𝗲𝗴𝗶𝗼𝗻𝗮𝗹 𝗖𝗼𝗺𝗽𝗶𝗹𝗮𝘁𝗶𝗼𝗻: Optimizes repeated nn.Module compilation, significantly reducing cold start times for LLMs. - 𝗧𝗼𝗿𝗰𝗵𝗜𝗻𝗱𝘂𝗰𝘁𝗼𝗿 𝗕𝗼𝗼𝘀𝘁: CPP backend enhances performance with FP16 support and max-autotune mode, outperforming eager mode in 97.5% of models. - 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗚𝗿𝗼𝘄𝘁𝗵: Introduction of projects like TorchRec and TorchFix expands - PyTorch’s versatility for AI applications. - 𝗜𝗻𝘁𝗲𝗹 𝗚𝗣𝗨 𝗦𝘂𝗽𝗽𝗼𝗿𝘁: Enhancements for Intel GPUs make PyTorch a more inclusive tool for diverse hardware setups. - 𝗕𝗮𝗰𝗸𝘄𝗮𝗿𝗱 𝗖𝗼𝗺𝗽𝗮𝘁𝗶𝗯𝗶𝗹𝗶𝘁𝘆: Smooth upgrade path from PyTorch 2.4, maintaining user workflows while optimizing performance. - 𝗙𝗹𝗲𝘅𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝗻 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗠𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺𝘀: New FlexAttention API and Compiled Autograd improve handling of attention in high-performance models. Article Authored by: Shalini

5

Founded in 2000 by🌟 Srikanth Velamakanni ,🔥 Pranay Agrawal and 🌟Ashwath Bhat ,🚀 Fractal offers artificial intelligence and advanced analytics solutions to Fortune 500 companies. 🚀🌟Fractal To File DRHP By September For $600 Mn IPO, Eyes $3 Bn Valuation🚀🌟 🌟Fractal's IPO Ambition: Fractal is set to file its Draft Red Herring Prospectus (DRHP) by September for a $600 million Initial Public Offering (IPO), targeting a $3 billion valuation. This bold move showcases the company's growth and vision for the future. 🌟Visionary Founders: Founded in 2000 by Srikanth Velamakanni, Pranay Agrawal, and Ashwath Bhat, Fractal has been at the forefront of artificial intelligence and advanced analytics solutions. Their dedication and foresight have propelled the company to serve Fortune 500 companies worldwide. 🌟AI and Analytics Leadership: Fractal's commitment to leveraging artificial intelligence and advanced analytics demonstrates its leadership in the tech industry. Their solutions empower businesses to make data-driven decisions, driving innovation and efficiency. 🌟Global Impact: With clients among the Fortune 500, Fractal's influence extends across various industries, proving the universal applicability and necessity of their cutting-edge solutions. Their journey inspires others to think big and strive for global impact. 🌟Perseverance and Growth: The founders' journey from starting Fractal in 2000 to preparing for a major IPO is a testament to their perseverance, strategic vision, and ability to navigate the evolving tech landscape. Their story encourages aspiring entrepreneurs to remain resilient and adaptable. 🌟Empowering Businesses: Fractal's advanced analytics solutions enable companies to unlock the full potential of their data, leading to smarter decision-making and enhanced business outcomes. This empowerment is a key driver of their success and a source of inspiration for other tech innovators. 🌟Strategic Milestone: The upcoming IPO is not just a financial milestone but a strategic move that will allow Fractal to scale further, innovate faster, and expand its market presence. It highlights the importance of strategic planning and execution in achieving long-term goals. 🌟Inspiring Future Entrepreneurs: Srikanth Velamakanni, Pranay Agrawal, and Ashwath Bhat's journey with Fractal serves as a powerful example for future entrepreneurs. Their success story demonstrates that with vision, dedication, and a focus on innovation, it's possible to achieve remarkable heights. ✅️Share Your Startup Story 🚀📲 https://lnkd.in/ggir3Qap ✅️visite our website🌐 https://lnkd.in/dad4nHP4 #Fractal #AI #AdvancedAnalytics #IPO #TechInnovation #SrikanthVelamakanni #PranayAgrawal #AshwathBhat #StartupSuccess #GlobalImpact #BusinessGrowth #Entrepreneurship #DataDriven #Fortune500 #Inspiration #Motivation #scoopearth

4

One more session alert by Vidvatta for people interested in prompt engineering. Topic - Efficient Prompt Engineering and Tuning for Language Models! 🗓️ Date: 26th May 2024 🕒 Time: 11 AM IST ⏳ Duration: 1 hour 30 mins 🔗 Register Here https://lnkd.in/gurGZmeV ✨ Join us in this webinar where you will have a unique experience of doing hands on together with mentor in this session and get live feedbacks when you are stuck. Are you interested in Large Language Models (LLMs) and their potential? Curious about how to leverage advanced prompting techniques for various tasks? Our upcoming webinar is perfect for you! ✨ What to Expect: A brief overview of LLMs and the importance of prompting. Prompting and Task Complexity- Classification of tasks: Easy, Medium, Hard. Advanced Prompting Techniques: * Zero-Shot and Few-Shot Learning: Definitions and applications. * Task Decomposition: Simplifying complex tasks. * Chain and Tree of Thought: Enhancing reasoning capabilities. Interactive Q&A Session: Open floor for attendee questions and sharing experiences. Future trends in LLMs and prompt engineering. Don't miss this opportunity to delve into the intricacies of LLMs and enhance your skills. Register now and be part of the conversation shaping the future of prompt engineering! In case of any queries contact [email protected] #Webinar #LLM #PromptEngineering #MachineLearning #ArtificialIntelligence #FutureTech #LearningCommunity

13

1 Comment

Happy to share that our paper, "UniMTS: Unified Pre-training for Motion Time Series", has been accepted to the Neural Information Processing Systems (NeurIPS) 2024. This is the first pre-trained model specifically designed for human motion time series, excelling in zero-shot activity recognition across a wide range of wearable devices, regardless of body placement or orientation. Our approach follows a CLIP-inspired pre-training framework, where we leverage contrastive learning to align motion data with LLM-enhanced activity descriptions. https://lnkd.in/gjJcVSiR #NeurIPS2024 #neurips #neurips2024 #neurips24 #MachineLearning

93

9 Comments

🚀 Are You Ready to Unlock the Full Potential of Small Language Models (SLMs)?🚀 Fine-tuning SLMs can significantly enhance their performance for specific tasks. Here’s an expanded approach, including tools and cloud services: 1. Dataset Preparation: Use high-quality, domain-specific datasets. Tools like Hugging Face's `datasets` library can help load and preprocess data efficiently. 2. Model Selection: Choose a pre-trained model such as Phi-3 or TinyLlama, accessible through libraries like Hugging Face Transformers. 3. Fine-Tuning Techniques: - LoRA: Fine-tunes smaller matrices to reduce parameters. - QLoRA: Combines quantization with LoRA for efficient memory usage. - DoRA/QDoRA: Further optimizations for resource-constrained environments. 4. Hyperparameter Tuning: Adjust learning rates, batch sizes, and other settings using tools like Optuna for automated optimization. 5. Cloud Services for Fine-Tuning: - AWS SageMaker: Offers powerful compute resources and simplifies the training process with built-in features. - Google Cloud Vertex AI: Provides a range of AI tools and customizable options for model training. - Microsoft Azure: Features NV-series VMs with NVIDIA GPUs and comprehensive ML lifecycle support via Azure Machine Learning. - Lambda Labs: Known for high-performance NVIDIA H100 Tensor Core GPUs and competitive pricing. - Oracle Cloud Infrastructure (OCI): Offers a Generative AI playground for fine-tuning models efficiently. 6. Evaluation and Iteration: After fine-tuning, evaluate model performance using relevant metrics and refine your approach based on results. By following these steps and utilizing these tools and cloud services, you can effectively fine-tune small LLMs to meet your specific needs! 💡 If you need further guidance, feel free to connect with Rohan Chikorde #ai #machinelearning #llm #finetuning #huggingface #cloudcomputing #generativeai #artificialintelligence #mlops #deeplearning #nlp #dataanalytics #bigdata #techtrends #innovation

2

Here is a very interesting question asked in Tredence Analytics. I have solved it with 3 methods. Link in the comments section. Here are the create inserts. CREATE TABLE cinema ( seat_id INT PRIMARY KEY, free int ); delete from cinema; INSERT INTO cinema (seat_id, free) VALUES (1, 1); INSERT INTO cinema (seat_id, free) VALUES (2, 0); INSERT INTO cinema (seat_id, free) VALUES (3, 1); INSERT INTO cinema (seat_id, free) VALUES (4, 1); INSERT INTO cinema (seat_id, free) VALUES (5, 1); INSERT INTO cinema (seat_id, free) VALUES (6, 0); INSERT INTO cinema (seat_id, free) VALUES (7, 1); INSERT INTO cinema (seat_id, free) VALUES (8, 1); INSERT INTO cinema (seat_id, free) VALUES (9, 0); INSERT INTO cinema (seat_id, free) VALUES (10, 1); INSERT INTO cinema (seat_id, free) VALUES (11, 0); INSERT INTO cinema (seat_id, free) VALUES (12, 1); INSERT INTO cinema (seat_id, free) VALUES (13, 0); INSERT INTO cinema (seat_id, free) VALUES (14, 1); INSERT INTO cinema (seat_id, free) VALUES (15, 1); INSERT INTO cinema (seat_id, free) VALUES (16, 0); INSERT INTO cinema (seat_id, free) VALUES (17, 1); INSERT INTO cinema (seat_id, free) VALUES (18, 1); INSERT INTO cinema (seat_id, free) VALUES (19, 1); INSERT INTO cinema (seat_id, free) VALUES (20, 1);

401

41 Comments

Many of ML practitioners might have heard the term "Extending context length" for LLMs. In case if you want to dive deeper into this topic along with its code implementation, give this blog a read. I have described about Position Interpolation, NTK aware and Dynamic NTK method, which are widely used for extending context length.

68

I have been asked by many #datascience , #aiml #professional and #enthusiast , What is the benefit of getting a Master Degree from premiere institute. I did my Master from Indian School of Business (AMPBA) and I feel fortunate to be the part of this Journey and I can list out some of the reasons below. 1. Quality of Education: Premier institutes are known for their excellent faculty, infrastructure, and academic resources. They provide high-quality education that equips students with the necessary knowledge, skills, and practical exposure required for their chosen profession. 2. Industry Recognition: Degrees and certifications from premier institutes are highly valued and recognized by employers and organizations. Graduates from such institutes are often sought after due to the rigorous training and qualifications they possess. 3. Networking Opportunities: Premier institutes offer excellent networking opportunities, allowing students to interact with industry professionals, attend seminars, and participate in events. These connections can be invaluable for future career prospects and professional growth. 4. Specialized Expertise: Premier institutes often have specialized programs or concentrations in specific fields, allowing students to gain in-depth knowledge and expertise in their area of interest. This specialized knowledge can give them a competitive edge in the job market. 5. Skill Development: Premier institutes emphasize not only theoretical knowledge but also practical skill development. They provide opportunities for internships, projects, and hands-on training, which help students develop essential skills required in their respective professions. 6. Personal Growth: The challenging academic environment and exposure to diverse perspectives at premier institutes foster personal growth, critical thinking, and problem-solving abilities, which are invaluable assets in any professional field. 7. Career Advancement: A degree or certification from a premier institute can open doors to better job opportunities, higher salaries, and faster career advancement. Employers often recognize the value of education from respected institutions. 8. Reputation and Prestige: Attending a premier institute carries a certain level of prestige and reputation, which can be beneficial for both personal and professional growth. It can enhance an individual's credibility and standing in their chosen field. While the investment in professional education from a premier institute may be substantial, the potential benefits in terms of knowledge, skills, networking opportunities, and career prospects often make it a worthwhile investment for individuals seeking professional growth and success. I am fortunate to meet professors like vijaya marisetty Subodha Kumar Manish Gupta Shailesh Kumar Niyati Chhaya to name a few. Thanks to BNY Mellon for sponsored me. Follow Bijan Dash for great insight and help. ISB Institute of Data Science (IIDS)

29

5 Comments

I am glad to share that our work on Geocoder for Q-Commerce has been accepted for publication in the Applied Data Science Track at the ACM CODS-COMAD, Dec 2024. Here's the abstract of the paper. "Accurate customer locations for quick (q)-commerce is a necessary requirement for under-30-minute online food delivery systems. Inaccurate locations can negatively impact delivery time promises, ranking of relevant restaurants and the accessibility of restaurants to customers. GPS locations captured using the customer smartphone application can be inaccurate due to the scattering of electromagnetic signals by vegetation, buildings or metallic reflectors, and manual error in placing the GPS pin on the application. Therefore, these incorrectly captured locations must be classified as inaccurate using a classifier and corrected using a geocoder. We design an end-to-end location correction system as a cascade of a location inaccuracy classifier and a geocoder. Apart from the address text and customer location inputs, we also use signals from the delivery partner's (DP) application. We propose a self-supervised multimodal architecture with text and numeric inputs for the location inaccuracy classifier that improves upon previous work on the ROC-AUC by 19 pp and the recall on an internal test set by 32 pp. For the geocoder, we evaluate sequence-to-sequence models and address matching architectures. We show that none of these architectures can individually meet the stringent accuracy targets for q-commerce in the Indian context due to sparse address input by customers, code-mixing, and the unstructured nature of Indian addresses. We ensemble these model outputs with historical delivery locations marked by DPs to report an under-65 m mean distance deviation from the delivery locations. The predicted locations are closer to the delivery locations by 74.7 m on average and 83\% of the predicted locations are closer to the delivery locations as compared to the customer-captured locations. Furthermore, we propose a relative benchmark to assess whether the geocoder accuracy meets the requirements for production-grade q-commerce systems, given the difficulty in precisely defining absolute accuracy benchmarks." Some of the paper's content is summarised in the two part-blog below. Part 1 on location inaccuracy classifier: https://lnkd.in/g65_3_vw Part 2 on geocoder for q-commerce: https://lnkd.in/gDdhrS65 Congratulations to the co-authors Yaswanth reddy Bytasandram Jose Mathew Tanya K. Sagar Jounkani cc. Jairaj Sathyanarayana #swiggy #swiggydatascience

130

2 Comments

It's interesting to see Deepinder Goyal's comments about this acquisition. He believes building a super brand (Zomato, Blinkit, and now District with Paytm Insider) is more important than building a super app (a Paytm-like aggregator). From a product perspective, this makes a lot of sense. You open a specific app for a specific action and don't get lost in the myriad choices an aggregator offers. The objective of doing one thing—and doing it well—always yields benefits. The larger trend of shifting focus from a simple Customer LTV to Portfolio LTV is prevalent across other domains too. Quite a few global mobile game companies have been trying to do the same. The mantra seems to be: provide different experiences through different interfaces, but tie the association to a larger brand. I wonder who else will follow suit. #zomato #blinkit #paytm

8

Want to perform hyperparameter tuning on your ML and DL algorithms? Learn about how you can do it with Optuna and neptune.ai

15

Good folks at Google and Harvard University have introduced FRAMES (Factuality, Retrieval, and Reasoning MEasurement Set), a comprehensive evaluation dataset designed to test RAG capabilities across factuality, retrieval accuracy, and reasoning. • 824 challenging multi-hop questions requiring integration of info from 2-15 Wikipedia articles • Tests 5 reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing • Rigorous quality checks, including temporal disambiguation and large output space • Baseline results with state-of-the-art LLMs like Gemini-Pro-1.5 and Gemma2 models • Novel multi-step retrieval pipeline using BM25 scoring • Best performance: 0.66 accuracy with (k,n,n_docs) = (5,5,10) configuration • Significant improvement over single-step 0.40 accuracy They discuss a unified framework for evaluating end-to-end RAG performance, addressing limitations in existing benchmarks that test components in isolation. Excited to see how this advances the development of more robust RAG systems! What are your thoughts on comprehensive RAG evaluation?

18

Perhaps the most critical thing in a Retrieval Augmented Generation (RAG) system is the retrieval or searching strategy. Poor retrieval will lead to irrelevant documents going to the LLM leading to irrelevant generated answers or worse hallucinations. Is cosine similarity everything? Do check out below illustration which talks about four of the most common strategies for playing around with retrieval strategies in RAG systems: 1. Similarity-based retrieval: This is based on embedding similarity, you can play around with different metrics like cosine, maximal margin relevance etc. 2. Similarity with Threshold retrieval: Useful where we cap the similarity threshold and can help in removing irrelevant document but can drop recall 3. Multi-query retrieval: Make the LLM generate variations of your query and get documents based on these multi-versions of your initial query, often useful to get more recall or coverage of documents 4. Reranker retrieval: Use a model specialised in ranking documents based on search relevance Often you can combine some of these techniques and even use ensemble retrievers. Do explore them and go beyond just cosine similarity!

17