kesavan sekar

🚀 Advancing Retrieval-Augmented Generation (RAG) with Hypothetical Document Embeddings (HyDE) 🔍 As we push the boundaries of information retrieval, Hypothetical Document Embeddings (HyDE) offers a novel method to enhance dense retrieval by injecting generative capabilities into the process. The two-step HyDE process revolutionizes how documents are embedded. Generative Document Creation: Using a large language model (LLM), HyDE generates a hypothetical document based on a user query, even if the content isn’t factually accurate. This method prioritizes capturing relevance, aligning the generated document with the query's intent. Contrastive Encoding: The generated document is passed through a contrastive encoder to create a vector representation that compresses the content and filters out hallucinated details, leaving a refined, semantically rich embedding. 💡 Why is this game-changing? HyDE simplifies dense retrieval by merging the generative power of LLMs with contrastive encoding, outpacing models like Contriever while performing on par with fine-tuned retrievers across multilingual tasks. What distinguishes HyDE from traditional RAG setups is the additional LLM-driven hypothetical generation step, making retrieval more robust and context-aware. I've been experimenting with this in a Llama-Index pipeline using groq and gpt hybrid model for generating HyDE embeddings and verifying RAG results to extend my pipeline's capabilities. The results are really staggering. Enhanced Query-Document Alignment: By using the data source context to update the user’s query, the LLM generates an optimized hypothetical document. This ensures the retrieval process remains stable, even when multiple reformulations of the same query are made. Think of it as reinforcing retrieval consistency across varying query forms. 💻 Implementation Insights: The process involves multiple LLM calls (2-3 depending on the use case), so careful planning is required for performance optimization. Balancing between local and remote LLMs is crucial for efficiency, with tools like Ollama, Groq helping mitigate inference time. This is just the beginning of how generative models and contrastive retrieval can coalesce to unlock new levels of relevance and efficiency in large-scale information systems. #AI #HyDE #LLMs #RAG #LlamaIndex #GPT #Groq #Ollama #LangChain #MachineLearning #DataRetrieval #InformationRetrieval #GenerativeAI

3

🚨ACL 2024 Outstanding Paper Award Alert 🚨 ➡️Paper Title: INDICLLMSUITE: A BLUEPRINT FOR CREATING PRE-TRAINING AND FINE-TUNING DATASETS FOR INDIAN LANGUAGES 🌟Few pointers from the paper 🎯Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. 👁In this paper Authors work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. 🎯Recognizing the importance of both data quality and quantity, their approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. 🎯They built a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. 🎯For instruction-fine tuning, they amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. 🎯Additionally, they addressed toxicity alignment by generating toxic prompts for multiple scenarios and then generated non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. 🎯 They hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. 🏢Organization: 🛎️Nilekani Centre at AI4Bhārat🛎️ Indian Institute of Technology, Madras 🛎️Microsoft🛎️Sant Longowal Institute of Engineering and Technology 🛎️Indian Institute of Information Technology Design & Manufacturing Kancheepuram, 🛎️Maharaja Agrasen Institute Of Technology, Delhi 🛎️Sarvam ,🛎️ NICT 🧙Paper Authors: Mohammed Safi Ur Rahman Khan, Priyam Mehta , Ananth Sankar , Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh Khapra 1️⃣Read the Full Paper here: https://lnkd.in/gqwQST-r 2️⃣Code: https://lnkd.in/giWPVntx 🎉 🎉  Congratulations to all the authors 🎉 🎉 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #ACL2024 #Indicllms #llms #llama2 #mixtral

16

🚀 Dive into the World of Apache Kafka! 🚀 Discover the Top 8 Apache Kafka Use Cases that are Revolutionizing Real-Time Data Processing! 💡 As a cutting-edge streaming platform, Apache Kafka unlocks a myriad of possibilities for crafting dynamic real-time data pipelines and applications. Whether you're a seasoned developer or just dipping your toes into the realm of streaming data, understanding these use cases is essential for staying ahead in the game. But wait, there's more! 🌟 🎉 BONUS Alert: Don't miss out on our exclusive FREE Live workshop where you can master the art of creating AI-powered apps using Kafka for real-time streaming data! 📊 📌 Secure your spot now: [Link to Register] 📌 Now, let's unravel the magic of Kafka with these 8 game-changing use cases: 1️⃣ Streaming Data: Harness Kafka's prowess as a central hub to seamlessly stream data from diverse sources like databases, applications, and IoT sensors, facilitating lightning-fast real-time data ingestion. 2️⃣ Log Aggregation: Witness Kafka's ability to collect and aggregate logs from myriad services, making them accessible in a standardized format for multiple consumers, revolutionizing monitoring and analysis. 3️⃣ Message Queuing: Empower loosely coupled producers and consumers to exchange messages effortlessly through Kafka's message queue topics, paving the way for asynchronous communication between microservices. 4️⃣ Data Replication: Experience Kafka Connect in action as it continuously replicates data between databases, ensuring seamless synchronization and supporting critical use cases like high availability and disaster recovery. 5️⃣ Monitoring & Alerting: Marvel at how Kafka transforms mundane service logs and metrics into actionable insights, enabling real-time alerts to be dispatched when anomalies are detected. 6️⃣ Change Data Capture (CDC): Witness Kafka's prowess in capturing change events from databases using CDC connectors, empowering applications to respond to data changes in real-time. 7️⃣ System Migration: Experience Kafka's magic as it facilitates seamless system migrations by decoupling old and new systems, enabling incremental functionality movement with minimal risk. 8️⃣ Real-Time Analytics: Explore how Kafka fuels real-time analytics by seamlessly collecting events to feed analytics systems and dashboards, delivering immediate insights on crucial business metrics. 💡 By unraveling these key use cases, you'll unlock a treasure trove of opportunities to leverage Kafka and craft innovative real-time applications and pipelines that propel your business forward! 👉 Which Kafka use case resonates with you the most? Share your thoughts and experiences below! Let's ignite a conversation! 🔥 Get ready to embark on an exhilarating journey into the realm of Apache Kafka! ✨ #ApacheKafka #RealTimeData #Innovation #DataStreaming #AI #Workshop

5

Read “Understanding Change Data Capture (CDC): A Guide to Real-Time Data Synchronization“ by Swayam Sampad Das on Medium: https://lnkd.in/ge5u-8QF

7

#PipelineAutomation in #ML involves creating #Automatedworkflows that handle the entire #ml process, from data ingestion to model deployment and monitoring. Here are some key components: #𝐃𝐚𝐭𝐚𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 Automating the collection and integration of raw data from various sources (databases, APIs, files, etc.). #DataPreprocessing Automatically cleaning, transforming, and normalizing data to prepare it for training. Handling missing values, scaling features, encoding categorical variables, etc. #FeatureEngineering Automatically creating and selecting features that improve model performance. Using domain knowledge, statistical techniques, or automated feature generation tools. #ModelTraining Automating the process of training machine learning models using predefined algorithms and hyperparameters. Implementing techniques like hyperparameter tuning and cross-validation. #ModelEvaluation Automatically evaluating models using various metrics (accuracy, precision, recall, F1 score, etc.). Comparing model performance and selecting the best-performing model. #ModelDeployment Automating the deployment of trained models to production environments. Ensuring models are accessible via APIs, web services, or other deployment mechanisms. #ModelMonitoring Continuously monitoring the performance of deployed models to detect issues like drift, degradation, or anomalies. Implementing alerting systems for proactive maintenance. #ModelRetraining Automating the process of retraining models periodically or when new data becomes available. Ensuring models remain accurate and up-to-date with the latest data. 𝑩𝒆𝒏𝒆𝒇𝒊𝒕𝒔 #Efficiency: Reduces manual effort and speeds up the machine learning lifecycle. Reproducibility: Ensures consistent and repeatable results by standardizing processes. #Scalability: Easily scales with increasing data volumes and computational demands. #Reliability: Minimizes human error and ensures robust, reliable workflows. #Agility: Enables rapid experimentation and iteration, facilitating faster model development and deployment. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 #Complexity Designing and maintaining automated pipelines can be complex and require expertise. #Integration Ensuring seamless integration between different tools and components can be challenging. #Monitoring & #Maintenance Continuously monitoring and maintaining automated pipelines require robust systems and processes. 𝐓𝐨𝐨𝐥𝐬 𝐚𝐧𝐝 𝐓𝐞𝐜𝐡𝐧𝐨𝐥𝐨𝐠𝐢𝐞𝐬 #ApacheAirflow #Kubeflow #MLflow #TensorFlowExtended (#TFX): #ApacheNiFi #DataRobot #MachineLearning #AIPipelines #MLAutomation #DataScience #ArtificialIntelligence #MLOps #TechInnovation #DataEngineering #AIWorkflow #BigData #DeepLearning #AIinBusiness #DataPipeline #TechTrends #ModelDeployment

11

1 Comment

I was shocked to see the below graph 🤯. I saw this Google Search trend graph on Chat GPT VS Perplexity VS Claude. I now feel like I am in a bubble surrounded by tech folks who use claude etc and to me it feels normal. But as you can see in the graph, the actual use of Claude & Perplexity is very very minute compared to Chat GPT. Especially Perplexity feels like its non-existent. And now a month back Chat GPT Search is also launched which will definitely take a share of this audience. Looking at this graph also makes me optimistic about building, It clearly shows that there is a huge gap in the market, and similar products can be built. What do you guys think ? #ai #startups #chatgpt #claude #perplexity

21

4 Comments

In Daniel Klitzke's latest article, he shows how you can use the Huggingface Transformers and Sentence Transformers libraries to boost you RAG pipelines using reranking models. #LLM #MachineLearning

23

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Comments

🔥 Decoding Thin Executor vs. Fat Executor in Apache Spark 🚀 Let's unravel these concepts within the context of Spark, shedding light on how they influence distributed data processing: 🔍 Thin Executor in Apache Spark: In Spark's ecosystem, a Thin Executor embodies a streamlined approach to distributed computation. These executors focus on efficiently executing tasks across a cluster while minimizing resource overhead. In Spark's architecture, Thin Executors typically involve lightweight task execution engines, such as the default TaskScheduler, which efficiently allocate tasks to available resources without significant additional functionalities. 💡 Key Benefits: Resource efficiency: Minimized overhead per executor, maximizing cluster utilization. Scalability: Easily scales to accommodate varying workloads and cluster sizes. Simplicity: Straightforward setup and management, ideal for general-purpose use cases. 💡 Challenges: Limited control: May lack advanced task scheduling and resource isolation capabilities. Performance tuning: Requires careful configuration to achieve optimal performance under diverse workloads. 🔍 Fat Executor in Apache Spark: On the flip side, a Fat Executor configuration in Apache Spark embraces a more feature-rich approach, incorporating additional functionalities for fine-grained control and optimization. This might involve using custom task schedulers, resource managers, or even containerization technologies to isolate and manage task execution environments more effectively. 💡 Key Benefits: Enhanced control: Provides advanced task scheduling and resource management capabilities. Optimization potential: Facilitates fine-tuning for specific workload characteristics and cluster configurations. 💡 Challenges: Increased overhead: Higher resource consumption per executor, potentially leading to reduced cluster utilization. Complexity: Requires expertise in configuring and managing custom scheduling and resource management components. Maintenance overhead: Introduces additional overhead for monitoring, troubleshooting, and updating custom components. In the realm of Apache Spark, the choice between Thin and Fat Executors hinges on the specific requirements and constraints of your data processing workload. While Thin Executors offer simplicity and resource efficiency, Fat Executors provide greater control and optimization potential at the cost of increased complexity and overhead. Understanding these trade-offs is crucial for architecting efficient and scalable Spark applications. #BigDataAnalytics #DataProcessing #DistributedSystems #DataScience #DataEngineering #ParallelComputing #ClusterComputing #ApacheHadoop #SparkProgramming #TaskScheduling #ResourceManagement #Containerization #DataOps #PerformanceTuning #DataManagement #CloudComputing #TechTrends #SoftwareDevelopment #DataArchitecture #DataDrivenDecisionMaking

10

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

💡 The hidden power of vision language models for efficient document retrieval. This paper introduces 𝐂𝐨𝐥𝐏𝐚𝐥𝐢, a novel approach to document retrieval that leverages the power of Vision Language Models (VLMs). 𝐖𝐡𝐲 𝐂𝐨𝐥𝐏𝐚𝐥𝐢 𝐒𝐭𝐚𝐧𝐝𝐬 𝐎𝐮𝐭: ✅ 𝐅𝐚𝐬𝐭𝐞𝐫 𝐚𝐧𝐝 𝐒𝐢𝐦𝐩𝐥𝐞𝐫: It processes document images directly, eliminating the need for slow and complex pre-processing steps like OCR and layout analysis. This translates to significant speed improvements (0.39 seconds/page vs. 7.22 seconds/page for standard retrieval). ✅ 𝐂𝐚𝐩𝐭𝐮𝐫𝐞𝐬 𝐌𝐨𝐫𝐞 𝐈𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧: By directly analyzing the image, ColPali can incorporate both textual and visual information within the document, leading to a more comprehensive understanding of the content. ✅ 𝐒𝐭𝐚𝐭𝐞-𝐨𝐟-𝐭𝐡𝐞-𝐀𝐫𝐭 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: ColPali achieves impressive results in document retrieval tasks while being significantly faster than traditional methods. ✅ 𝐈𝐧𝐭𝐞𝐫𝐩𝐫𝐞𝐭𝐚𝐛𝐢𝐥𝐢𝐭𝐲: The model offers some level of explainability, helping users understand why specific documents are retrieved for a query. Key to ColPali's success is its "late interaction" mechanism. Instead of directly comparing textual and visual embeddings, it feeds them into a separate scoring layer. This layer considers both individual pieces of information and how they relate to each other, leading to a more nuanced understanding of the document's relevance. 🔗 Demo: https://lnkd.in/eYapwP7G 🔗 Model: https://lnkd.in/eGhD3yDF 🔗 Paper: https://lnkd.in/eQzhWSMZ Want to learn more? ↓ Follow for more content cutting-ednge AI content!

6

Euler's Totient Function Euler’s Totient function Φ (n) for an input n is the count of numbers in {1, 2, 3, …, n-1} that are relatively prime to n, i.e., the numbers whose GCD (Greatest Common Divisor) with n is 1 . Thanks to Sandeep Jain for this informative session on Euler Totient Function in Mathematics for Competitive Programming and all GeeksforGeeks Team . #softwareengineer #dsa #interviews #maths #sde #programming #c++ #linkedin #rahulverma

11

1 Comment

***Optimizing Parquet File Processing in S3 – Efficient Removal of Duplicate Schemas***   When handling large datasets stored in Parquet format within S3, managing schemas becomes crucial for ensuring efficient data processing. Multiple Parquet files often contain embedded metadata, and redundant schema loading can slow down processing. Here’s how I recently addressed the challenge of removing duplicate schemas during data processing:   **Scenario:** We had several Parquet files stored in an S3 bucket, each containing schema metadata. To avoid duplicate schema processing, we needed to ensure that the schema was deduplicated during the ingestion phase.   **Solution:** 1. **Read Parquet Files with Schema Merging:**    Using Spark, I read the Parquet files and enabled schema merging to handle any schema evolution that occurred across different files.    ```python    df = spark.read.option("mergeSchema", "true").parquet("s3://your-bucket-name/path/to/parquet/files")    ```   2. **Deduplicate Data:**    To prevent processing duplicate rows, I applied the `distinct()` transformation on the DataFrame to remove any redundant records that could impact the output.    ```python    deduplicated_df = df.distinct()    ```   3. **Write Clean Data Back:**    Finally, the deduplicated data was written back to S3 in a clean and optimized format, ensuring no schema duplication and efficient querying.    ```python    deduplicated_df.write.parquet("s3://your-bucket-name/path/to/output")    ```   **Why This Matters:** Avoiding duplicate schemas and redundant data processing helps save compute resources and improve the overall performance of data pipelines. With schema-on-read in Parquet, ensuring proper schema management is essential for scalability as your data evolves.   **Pro Tip:**  Using Spark’s `mergeSchema` option is crucial when working with evolving schema versions across different Parquet files.   Have you encountered challenges with schema duplication or data evolution in your pipelines? Let’s discuss best practices! Follow Swarnali S. #DataEngineering #ApacheSpark #BigData #ParquetFiles #AWS #S3 #Optimization #DataProcessing #SparkTips #CloudComputing

59

1 Comment

🚨 New video alert! DSPy: machine learning attitude towards prompting🚨 LLMs have fatigued me, it's so easy to create a good demo and so hard for evaluation and productanization. That's a poisonous mix for a data-scientist. DSPy can help you elevate this pain, by treating the generative stack as if it was a neural network, with its own parameters. Composing different patterns one on top of another, evaluating, optimizing. We don't have to reinvent the wheel each time, rephrasing again and again CoT pattern, Karpathy prayer's, few shot examples and so forth. In the video you can watch a tutorial that is not too simple and not too complex that can simulate how to use this framework, by implementing a customer service LLM based intent classifier. Personal note - This is the first video that wasn't born out of my streaming experience. Meaning it is more polished than my previous videos. Technical note - DSPy is great

18

1 Comment

llama-3 has launched 🔥🔥🔥. - 2 Models : 8B and 70B models - They were trained for 1.3M and 6.4M GPU hours respectively - the 70B model, Outperforms GPT-3.5 on curated 1200 prompt evals - Context length is 8k tokens Exciting times ahead with Meta. Link - https://lnkd.in/dQJZupZq #startups #AI

17

2 Comments