Shiva Kumar, PhD

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

Why Parquet Alone Is Not Enough: Iceberg & Hudi Explained with Example 🚀 🔍 Managing data lakes is hard. What happens when your datasets; - Evolve, - Need real-time updates, - or Require historical audits? Let’s explore this with a simple example and how Apache Iceberg and Apache Hudi enhance your data lake game. #Example: E-Commerce Sales Data Imagine you’re managing daily sales transactions for an e-commerce platform. #Goal’s: • Add a new column (campaign_id) to track promotions. • Debug a pricing error by querying historical data. • Update invalid records efficiently. #With_Parquet_Alone: 1. Adding campaign_id: You must rewrite all existing files, incurring time and cost. 2. Debugging historical data: You need manual backups or external versioning tools. 3. Updating invalid records: Entire files must be rewritten due to a lack of transactional support. #With_Iceberg_or_Hudi: 1. Adding campaign_id: Schema evolution allows you to add columns dynamically, without rewrites. 2. Debugging historical data: Use time travel to query the dataset before the error: SELECT * FROM sales_table FOR TIMESTAMP AS OF '2024-11-15'; 3. Updating invalid records: ACID transactions enable fast, reliable updates with simple SQL: MERGE INTO sales_table USING updates ON sales_table.order_id = updates.order_id WHEN MATCHED THEN UPDATE SET price = updates.price; #Result: Simplified operations, faster queries, and scalable data management. #Benefits of Iceberg & Hudi Over Parquet Both Iceberg and Hudi extend Parquet with: • Schema Evolution: Add, drop, or rename columns without rewriting files. • Time Travel: Query data as it existed at any point in time. • ACID Transactions: Perform reliable updates, deletes, and concurrent writes. • Partitioning Without Hassle: • Iceberg: Hidden partitions with automatic pruning. • Hudi: Optimized for upserts with built-in indexing. • Real-Time Support: • Hudi: Designed for streaming and CDC (Change Data Capture). • Iceberg: Best for batch analytics. • Metadata-Driven Optimization: Faster queries via centralized metadata. #Limitations of Parquet Alone: • No schema evolution. • No ACID transactions for updates or deletes. • Manual partitioning is error-prone. • No time travel for historical queries. • Limited real-time support for streaming or CDC. #Takeaway -If your data lake needs dynamic schemas, real-time updates, or historical insights, table formats like #Iceberg and #Hudi are game changers. They turn your static #Parquet files into a powerful data management system. Ready to make the switch? Let’s discuss how these formats can fit your data lake! 💬

55

10 Comments

“𝐉𝐮𝐩𝐲𝐭𝐞𝐫 𝐧𝐨𝐭𝐞𝐛𝐨𝐨𝐤𝐬 𝐬𝐩𝐚𝐫𝐤 𝐭𝐡𝐞 𝐜𝐫𝐞𝐚𝐭𝐢𝐯𝐢𝐭𝐲 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞, 𝐛𝐮𝐭 𝐌𝐋𝐎𝐩𝐬 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤𝐬 𝐢𝐠𝐧𝐢𝐭𝐞 𝐭𝐡𝐞 𝐟𝐥𝐚𝐦𝐞𝐬 𝐨𝐟 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧-𝐠𝐫𝐚𝐝𝐞 𝐬𝐮𝐜𝐜𝐞𝐬𝐬.” Hello everyone, In this blog, I explained: 1. How jupyter notebooks aid data exploration, but MLOps frameworks are crucial for scaling Data Science projects and ML models, ensuring reliability, scalability, and reproducibility. 2. How to move your existing jupyter-notebook to a production-grade Data Science project by modularizing your codebase and using several MLOps tools. Please feel free to drop your comments/suggestions. Love. Peace. Aashish #datascience #mlops #ai

23

Heterogeneous Graphs and Relational Graph Convolutional Networks (RGCNs): How can we prevent overfitting? In my post of last week [https://lnkd.in/d5cZn5ur], I spoke about Heterogeneous Graphs - examples of Heterogeneous Graphs being Knowledge Graphs which might have several node types or relationship types. I had mentioned that training such complex networks can become computationally intensive because each relationship type has to be associated with different weight/parameter thus increasing the number of  overall parameters to be learnt for a RGCN model and thus resulting in overfitting of the model. I had then highlighted about "Block Diagonal Matrices" And "Basis Learning" to reduce the number of parameters of a RGCN model. Let us try and put some more insight into "Block Diagonal and Basis Learning" and understand what these actually mean! Block Diagonal Matrices: The key idea with "Block Diagonal Matrices" is to make the Transformation matrices corresponding Wr as sparse as possible- that is: to enforce a block diagonal structure as shown in the Figure 1 below so that the non-zero elements lie only on the specific blocks of the transformation matrix "Wr". This reduces the number of non-zero elements which means that during computation you will have to only estimate the non-zero elements. This reduces the training time considerably What do you lose by using Block Diagonal Matrices? Well, we do lose some information by using Block Diagonal Matrices - the embedding dimension far from one another will not be able to interact with each other It’s only the embedding dimension that are along the non-zero entities that will be able to interact with each other that is: only the neurons along the non-zero entities will be able to interact with each other Basis Learning: The second idea to make the RCNs more scalable is to share the weights amongst the relationship types. That is: we do not want the relationships to be independent of each other, but we want the relations to share the information based on some magnitude of weighting. This can be achieved by elegantly representing the Transformation matrices Wr as a linear combination of "Basis" matrices. These "Basis" matrices are also termed "Dictionaries" In this approach, the weight matrix Wr will be represented as a linear combination i.e. a weighted sum of Dictionary matrices Vb as highlighted in the Equation below In the equation, arb is the "importance" weight corresponding to the relation for the matrix "b". Thus, every relation will have to learn its importance weight for each of the basis matrices. This significantly reduces the number of parameters - as the basis matrices may be 10 or a larger number whereas number of relation types may be thousands and arb are scalar numbers will have to be learnt #graphs #machinelearning #deeplearning My Technical Blogs Library 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on Engineering

6

CLASSES in SCALA are pretty much like CLASSES in C++ & Other OBJECT ORIENTED PROGRAMMING LANGUAGE. #SCALA #BIGDATA #CPP #C++ Karthik K. object CLASS { class Point{ private var _x = 0 private var _y = 0 private val bound = 100 def x: Int ={ return _x} 👈 //GETTER for private var _x def y: Int ={ return _y} 👈 //GETTER for private var _y def coord( x:Int, y:Int)={ 👈 //SETTER for private vars _x & _y this._x = x 👆👇 /"This" keyword store address of the object calling the function this._y = y println(this._x,this._y) } } def main(args: Array[String]):Unit = { val a = new Point 👈 //Object creation of Point in Heap Memory a.coord(20,30) print(a) } } OUTPUT:

4

3 Comments

Understanding Narrow and Wide Transformations in Apache Spark In Apache Spark, transformations play a critical role in defining how data is processed across distributed clusters. These transformations can be broadly classified into narrow transformations and wide transformations, each influencing the efficiency and data flow within Spark applications. 1. Narrow Transformations: A narrow transformation is one in which each output partition depends only on a single input partition. Since the data required to generate an output partition resides in the same partition as the input data, Spark can process these transformations without requiring data to move across nodes. This minimizes network traffic and improves performance. Examples of Narrow Transformations: • map(): Applies a function to each element of the RDD/DataFrame without requiring data exchange between partitions. • filter(): Selects elements based on a condition, retaining them within the same partition. • flatMap(): Similar to map(), but can generate multiple elements from a single input element, still preserving the partition boundaries. 2. Wide Transformations: Wide transformations involve operations where the output partition depends on data from multiple input partitions, often triggering a shuffle—a costly operation where data is exchanged across the cluster. These transformations require coordination between nodes, leading to network I/O, which can impact performance. Examples of Wide Transformations: • groupByKey(): Groups data by key, causing a shuffle as all values associated with the same key need to be aggregated across partitions. • reduceByKey(): Similar to groupByKey(), but applies a reducing function during the shuffle phase to optimize data movement. • join(): Combines data from two datasets based on a key, triggering a shuffle to align matching keys across partitions. Key Difference: Data Shuffling The fundamental difference between narrow and wide transformations lies in data shuffling. In narrow transformations, Spark processes data locally without any network overhead. However, in wide transformations, the data must be reshuffled across partitions, leading to increased computational and network costs. For example, a map() operation applies functions independently within partitions, while a reduceByKey() requires the data to be shuffled so that values for the same key are processed together. Follow me(Raghu Raman A V) for more articles on Big Data, Data Engineering, Azure and cloud ! #BigData #ApacheSpark #DataEngineering

15

1 Comment

🚀 Have you heard of tech-first and tech-enabled companies? Say, Google,Tesla & Walmart,Pfizer respectively - Understanding the distinction is crucial in today’s rapidly evolving business landscape. ⚠️ Caution: The percentages provided are approximations based on my experience in software & data engineering and may vary significantly across different organizations and industries. Share your insights or corrections in the comments. CIO: In tech-first companies (40%), the CIO has strategic oversight, focusing on innovation. In tech-enabled companies (90%), they drive technology adoption and integration for operational efficiency. CTO: Tech-first companies (90%) see the CTO central to tech innovation, while in tech-enabled companies (40%), the focus shifts to integrating technology rather than core development. CDO: In tech-first firms (80%), the CDO innovates data strategies; in tech-enabled companies (60%), they align data strategy with business goals. Data Product Manager: Tech-first (75%) focuses on technology-driven solutions, while tech-enabled (60%) ensures data products meet user needs. Business Analyst: In tech-first companies (60%), they leverage data for new opportunities, while tech-enabled (75%) integrates data into business processes. Data Analyst: Tech-first (70%) emphasizes tech-driven analysis for initiatives, while tech-enabled (80%) focuses on extracting insights for reporting. Pricing Manager: They optimize strategies in tech-enabled (75%) firms and develop dynamic models in tech-first (60%) companies. Data Architect: In tech-first (80%) companies, they create innovative solutions; in tech-enabled (60%), they design data systems. Analytical Translator: Tech-first (75%) firms rely on them to translate insights into strategies, while tech-enabled (65%) firms bridge technical and non-technical teams. Analytics Engineer: They innovate solutions in tech-first (80%) companies, while in tech-enabled (55%), they build analytics infrastructure. Decision Scientist: Their role is more pronounced in tech-first (80%) companies, employing data for strategic decisions, while in tech-enabled (70%), they also use data effectively. Machine Learning Engineer: In tech-first companies (85%), they focus on developing advanced solutions; in tech-enabled firms (50%), they implement models. Applied Data Scientist: Tech-first (85%) firms use them for advanced methodologies; in tech-enabled (60%), they apply algorithms with domain knowledge. Data Scientist: They are deeply integrated in tech-first (90%) companies for product development, while in tech-enabled (10%), they specialize in analysis. Data Engineer: In tech-first (80%), they innovate data systems; in tech-enabled (70%), they maintain data pipelines. Data Governance Officer: They ensure data compliance in both settings (tech-first 75%, tech-enabled 65%). Data Quality Analyst: They focus on data quality in tech-first (80%) and maintain integrity in tech-enabled companies (70%).

27

6 Comments

❝ How do you taste your Wine? With the right Glass, served from the right Bottle. How do you utilize your LLM? With the right Task, served from the right Compute environment. ❞ If the quote above caught your interest, check out my blog post that was featured on the Toyota Connected India Medium channel last week: Leveraging LLMs in Cloud - 🔗 https://lnkd.in/gfHsdevH --- I hope this blog adds a bit of amusement for the seasoned AI/ML engineers 🧠👨‍💻 with the carefully crafted images and perhaps even the silly wine analogies 😉. Whether you're a cloud engineer ☁️ 👨‍💻, a product owner of cloud-based apps, or someone new to Generative AI in the cloud, this blog offers a detailed, beginner-friendly look into the selection and deployment of LLMs in cloud environments. --- A heartfelt thanks 🙏 to my colleagues at Toyota Connected India for taking the time to review, shape this piece and even share a creative writeup like below in the original post 🫡 👇 🤔 Curious to dive in? Click the link below to explore the blog on Medium! 🌐 🔗 https://lnkd.in/gfHsdevH

18

Recently one of my mentee asked "We study a lot of unsupervised techniques like - Clustering, Dimensionality reduction techniques. But I have rarely seen these models ever used outside of exploratory data analysis, Are they ever used in an actual modeling tasks powering some real world applications?" The simple answer to this is yes. But setting up an unsupervised problem is not that straightforward and often tricky. Here are some of the areas where unsupervised learning are quite effective: Anomaly Detection: Unsupervised learning can help detect outliers or anomalies in data, especially in situations where labeled data is scarce. For instance, in fraud detection, the model like Isolation Forest, DBSCAN, One-Class SVM, Inverse distance weighting etc are used to identify unusual patterns of activity that do not conform to typical user behavior. Recommendation Engine: Dimesionality Reduction techniques like SVD, non negative Matrix factorization etc are often used to model user, items and calcaulate user-item preference - SVD, Non Negative Matrix factorization, PCA, Factorization Machine etc User segmentation and Market Basket Analysis - Understanding user segments and Clustering and association rule learning can be applied to identify patterns in purchasing behavior, such as which products are often bought together. This helps in inventory management, product bundling, and personalized promotions. NLP and Computer Vision : Unsupervised learning is used in NLP tasks to discover topics in a corpus of text (e.g., Latent Dirichlet Allocation), cluster documents, or reduce the dimensionality of word embeddings for better performance in downstream tasks. Autoencoders and other dimensionality reduction methods are used to compress images and remove noise, making it easier to store or analyze high-dimensional image data. The biggest challenge of unsupervised modeling is evaluation: In absence of a true label how would you measure if the model trained is effective? Unsupervised metrics like cluster distance or goodness of fit are usually very difficult to tie to a business metric and make sense of how good the model actually is. In such scenarios one needs to come up with sensible assumptions or techniques to link the modeling outcome to some business metrics and evaluate the model. In most of the scenarios we tend to convert a problem without access to label in to supervised problem by creating proxy label. For example in video recommendation if you don't have ratings from user, you can use duration watched, or click to create a proxy label for user preference. The biggest example of such technique is Unsupervised Pre-training which is in the heart of why LLMs like Bert, GPT-2 etc so effective. These models are trained using unsupervised pre-training tasks like masked token prediction, next sentence prediction or next token prediction, while the label is crafted from the text corpus itself and doesn't need any explicit labeling.

30

4 Comments

Text chunks in retrieval may risk losing context. The below article introduces contextualizing each chunk using an LLM. The added cost and latency required for contextualizing is partially addressed through a lightweight LLM (haiku) and prompt caching. Along with other known techniques (BM25, reranking, etc.), a significant reduction in retrieval failure is achieved. Overall, a good article to read. https://lnkd.in/g_5fgAN8

10

Comment: Around 350 BC an Indian Sanskrit Scholar built a rule-based analog algorithm for in effect an ancient language machine. Title: PhD student solves a mysterious ancient Sanskrit text algorithm after 2,500 years. See… https://lnkd.in/dfupfx6y And… https://lnkd.in/end3UYpe Around 350 BC, the scholar Pāṇini composed a grammatical treatise of astounding breadth and comprehension. Known as the Aṣṭādhyāyī, it contained 4,000 sūtras, or rules, for writing classical Sanskrit. In effect the rules necessary to construct an ancient language machine Pāṇini’s treatise isn’t meant to just describe classical Sanskrit or prescribe usage like a fussy language arts teacher. It’s designed to help its readers generate “flawless” Sanskrit. He built a system of operations that allowed a reader to combine base words and affixes to generate proper word forms. The user could then combine those words to construct impeccable sentences. All readers had to do was follow the instructions. Effectively, he created an analog algorithm that would produce the same Sanskrit word, sentence, and hymn each time — preserving the language and settling disputes over correct usage for time immemorial. Rishi Rajpopat a University of Cambridge Ph.D. student, may have solved the mystery of how to use Pāṇini’s language machine accurately. While working on his thesis, Rajpopat decided that this meta morass could not have been what Pāṇini intended with the Aṣṭādhyāyī. “Pāṇini had an extraordinary mind and he built a machine unrivaled in human history. He didn’t expect us to add new ideas to his rules. The more we fiddle with Pāṇini’s grammar, the more it eludes us,” Rajpopat said in a press release. Rajpopat returned to Pāṇini’s original meta-rule and considered it with a fresh lens. When he found a rule conflict, rather than give preference to the later rule in serial order, he deferred to the word that came later in the sentence. Because Sanskrit is written left to right, that meant applying the rule appropriate to the word on the right. Rajpopat’s research looked at many other rule conflicts, and thorough textual evidence shows that the meta-rule results in the correct form time and again. This potentially makes it possible to construct millions of grammatically correct classical Sanskrit texts. H/T Moshe Vardi

64

8 Comments

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Comments

Converting a list to string, with a delimiter. Implementation of inbuilt mkString() function in SCALA. #SCALA #SCALADEVELOPER #BIGDATADEVELOPER #BIGDATA Karthik K. def main(args: Array[String]): Unit ={ //var a = "Tanuj" //lowerCase(a) var s = "" var l = List(1,2,3,4,5,6,7,8) for( i <- 0 until l.length) { if( i == l.length -1) { s = s.concat(l(i).toString) } else { s = s.concat(l(i).toString) s = s.concat(':'.toString) } } print("The elements of list when converted to string are:-" + s) } } OUTPUT:

2

Quantized embeddings offer significant advantages for NLP models, including reduced storage requirements, faster computations, lower energy consumption, and improved model interpretability. They are becoming increasingly important for enabling efficient and scalable NLP on resource-constrained devices and cloud-based applications. Read my latest article below

6

TensorFlow vs. PyTorch: Finding the best Deep Learning framework for your AI journey 🚀 Which framework will boost your career goals? Whether you’re excited about diving into cutting-edge research with PyTorch or eager to create scalable AI models with TensorFlow, each option has its own perks! From how easy they are to use and the support you can get from the community to their flexibility and scalability, understanding these differences can really set you up for success in the fast-paced world of AI. I think Juliette Carreiro's article for @ironhack is a fantastic resource for anyone starting their journey in Data Science and Machine Learning! On another note, I’m still looking for a German-speaking Data Manager to join a leading company on an exciting automotive project in Frankfurt. If this sounds like a good fit for you or someone you know, please reach out here or at [email protected]. Happy to share more details! #DeepLearning #AI #DataScience #MachineLearning #TensorFlow #PyTorch #CareerInTech #Hiring #ironhack #redsofa #datamanager

12

1 Comment

𝐎𝐧𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐛𝐢𝐠𝐠𝐞𝐬𝐭 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 𝐢𝐧 𝐚𝐝𝐯𝐚𝐧𝐜𝐢𝐧𝐠 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐢𝐬 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬 𝐭𝐡𝐚𝐭 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐟𝐫𝐞𝐞-𝐟𝐨𝐫𝐦 𝐭𝐞𝐱𝐭. 𝘐𝘮𝘢𝘨𝘪𝘯𝘦 𝘢 𝘮𝘶𝘭𝘵𝘪-𝘢𝘨𝘦𝘯𝘵 𝘴𝘺𝘴𝘵𝘦𝘮 𝘸𝘪𝘵𝘩 𝘢 𝘳𝘦𝘴𝘦𝘢𝘳𝘤𝘩𝘦𝘳 𝘢𝘯𝘥 𝘢 𝘸𝘳𝘪𝘵𝘦𝘳 — 𝘸𝘰𝘶𝘭𝘥 𝘢𝘥𝘥𝘪𝘯𝘨 𝘢 𝘧𝘢𝘤𝘵-𝘤𝘩𝘦𝘤𝘬𝘪𝘯𝘨 𝘢𝘨𝘦𝘯𝘵 𝘪𝘮𝘱𝘳𝘰𝘷𝘦 𝘳𝘦𝘴𝘶𝘭𝘵𝘴? 𝘞𝘪𝘵𝘩𝘰𝘶𝘵 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘦𝘷𝘢𝘭𝘶𝘢𝘵𝘪𝘰𝘯 𝘮𝘦𝘵𝘩𝘰𝘥𝘴, 𝘪𝘵’𝘴 𝘥𝘪𝘧𝘧𝘪𝘤𝘶𝘭𝘵 𝘵𝘰 𝘮𝘦𝘢𝘴𝘶𝘳𝘦 𝘵𝘩𝘦 𝘪𝘮𝘱𝘢𝘤𝘵 𝘰𝘧 𝘴𝘶𝘤𝘩 𝘤𝘩𝘢𝘯𝘨𝘦𝘴. While standardized tests like MMLU and HumanEval exist for general-purpose models, they have limitations like data leakage. Evaluating custom AI applications is even tougher. If you're tackling this evaluation problem or have insights on improving AI testing frameworks, I’d love to hear your thoughts!  #AI #GenerativeAI #AIInnovation #LLMs #AIResearch

18

Understanding LangChain Wrappers for Retrieval For detailed sample code, check out the Google Colab notebook(https://lnkd.in/g5NsYFwH) LangChain Wrappers are powerful tools that simplify the retrieval process by providing seamless access to external data sources like Wikipedia, news, and web searches. These Wrappers abstract away the complexities of data retrieval, allowing developers to focus on high-level logic while improving the efficiency and accuracy of applications. By using Wrappers in LangChain, developers can build sophisticated retrieval-augmented generation (RAG) systems without worrying about complex integrations. The result is a streamlined, consistent, and highly effective method for retrieving and utilizing relevant data in AI models. #LangChain #AI #MachineLearning #DataRetrieval #RAG #NLP #AIDevelopment #Technology #InformationRetrieval #ArtificialIntelligence

49