Aleksandr Tynianov’s Post

"Ultimately, the end user isn’t concerned about the intricacies of whether it’s a vector search, keyword search, rule-driven search or even a “phone a friend” search. What matters most to the user is getting the right answer. Rarely does this come from relying solely on one methodology. Understand your use case and validate your test scenarios … and… don’t be lured by shiny objects just because they’re popular." https://lnkd.in/e3TxBs8M #heroku #salesforceai #ai

#LLM can help you simplify using #data to get simple and usable analytics. In this article, Daniel Ong demonstrates how to build a great Semantic search using Snowflake Cortex on top of a dataset of Bright Data, containing #webdata from the AirBnB website. Example of using #webdata for the best #ai and #analytics, leveraging the Snowflake-BrightData partnership. Michael Rhodes Félix Salacroup Judit Daniel Pascal Ferro Tony Young https://lnkd.in/dw_c4mxa

Tired of drowning in a sea of irrelevant search results? The struggle is real. Check out our latest blog exploring the frustrations of ineffective searches and how #AI-powered Enterprise Search is revolutionizing knowledge retrieval. Say goodbye to outdated data and hello to efficiency! https://lnkd.in/gBWXYXJD #FutureTech #AISearch #AIInnovation #KnowledgeDrivenWorkspaces #EnterpriseSearch #Productivity #Innovation #AI #EnterpriseSearch #KnowledgeManagement Preethy Raghu Asokan Ashok Balaji Ramachandran Brian Friedman Ashvin Lima Ramanathan Sivasubramaniam

🚀 6. Understanding Retrievers in detail-LangChain Retriever Methods 🚀 I'm excited to share the 6th article in my Retrieval-Augmented Generation (RAG) series! This time, we're diving into the world of retrieving data from vector stores and the powerful concepts of Semantic Search and Maximum Marginal Relevance (MMR). 🤖🔍 💡 What's covered in this article: 🛠️ How to Use a Vector Store as a Retriever: Learn how retrievers fetch the most relevant information from your vector store to power your AI systems. 🔍 What is Semantic Search?: Discover how semantic search focuses on finding meaning, not just matching keywords. 🔄 What is Maximum Marginal Relevance (MMR)?: Understand how MMR ensures that your search results are relevant and diverse, avoiding repetition. ⚖️ Semantic Search vs. MMR: When to use each method? I explain which one is best depending on whether you need purely relevant results or a mix of relevance and variety. ❓ Questions Answered: 1. What exactly is a vector store retriever, and why is it important? 2. How does semantic search improve the way we find information? 3. Why is MMR useful for avoiding redundant search results? 4. When should you use semantic search vs. MMR? 🎯 Why this matters: This article will help you improve the way you build AI-powered search and question-answering systems by optimizing how you retrieve data. Whether you're working on summarization, chatbots, or any other RAG application, these concepts will take your work to the next level. 📈✨ 💡 Curious to learn more? Check out the full article here! 👇 https://lnkd.in/e6KpRWJV #RAG #MachineLearning #AI #LangChain #MMR #SemanticSearch #AItools #generativeai This post balances useful information with clear bullet points, making it engaging and beneficial for readers to understand why the article is valuable. Let me know if you need adjustments! 😊 Here is the link for RAG series https://lnkd.in/egV_7rme

We're halfway through 2024, and no AGI yet! But it looks like we might skip AGI and go straight to SSI 😉 Until the next AI buzzword emerges, here's what really matters. Two main concepts drive the form factor of LLM applications: 1️⃣ Retrieval Augmented Generation (RAG)  2️⃣ Agentic Systems Combine the two concepts driving AI Applications today, and you get: 🥁🥁🥁🥁🥁🥁🥁🥁 🌟Agentic RAG🌟 It combines RAG's information retrieval paradigm with an AI agent's ability to use tools, reason, and plan. You can think of it as a 2-for-1 bundle deal on building AI applications. To understand Agentic RAG,Imagine an AI that can: 1️⃣ Decompose complex tasks 2️⃣ Choose the right tools for each sub-task 3️⃣ Decide when to retrieve information vs. use its own knowledge 4️⃣ Execute multi-step problem-solving strategies Now you can stop imaging, as we've just released an in-depth guide on implementing Agentic RAG using: ↦ Anthropic's Claude 3.5 Sonnet for the Agent's brain 🧠 ↦ LlamaIndex for building the Agent integration 👷🏾♀️ ↦ MongoDB as the memory provider 💽 Here's what you'll learn: ✅ Build a full Agentic RAG pipeline ✅ Optimize embedding processes ✅ Create AI agents with advanced reasoning skills ✅ Leverage MongoDB for efficient knowledge retrieval Tutorial: https://lnkd.in/ehgrUycR Code: https://lnkd.in/gR3bqyjx #artificialintelligence #genai #ai #agenticsystems #aiagents #mongodb #claude

Generative AI & Traditional Search Engines Working Together The future of search is more than just finding information—it’s about understanding context, providing insights, and delivering results that make sense. With Generative AI and powerful search engines like Elasticsearch, Solr, and OpenSearch, we have the technology to take search experiences to new heights. Here’s how each tool brings unique strengths to the table: Generative AI is transforming search by understanding context and generating responses in natural language. It’s ideal for answering complex questions, summarizing information, and offering a more conversational search experience. Meanwhile, traditional search engines excel in areas that require speed, precision, and real-time data handling. Here’s how they stack up: - Elasticsearch: Known for its lightning-fast retrieval and real-time indexing, Elasticsearch is a top choice for environments where performance and scale matter, such as monitoring systems and e-commerce.    - Solr: As an Apache project, Solr provides highly customizable indexing and query capabilities. It’s a great option for complex data structures and offers strong performance in environments where exact matching and custom queries are needed.    - OpenSearch: Originally derived from Elasticsearch, OpenSearch maintains a focus on scalability and flexibility, making it a solid choice for applications needing open-source, highly customizable search options. Combined Power: Hybrid Search Solutions These tools, combined with Generative AI, offer a comprehensive solution for both structured and unstructured data needs. Imagine Elasticsearch or Solr retrieving relevant data, with Generative AI shaping those results into meaningful insights. This hybrid approach allows us to leverage the speed and precision of search engines alongside the context and conversational ability of AI. The result? A search experience that’s fast, insightful, and adaptive to user needs. It’s not about choosing one tool over another—it’s about integrating these technologies to deliver search that’s truly impactful. I’d love to hear others' thoughts on this approach and learn about any research that explores the potential of combining these tools. What do you think? #SearchTechnology #GenerativeAI #Elasticsearch #Solr #OpenSearch #Innovation #BigData #ESRE

Is Search Dead Now That AI Is Here? Short Answer: No! Long Answer: Search, as we knew it, is definitely evolving. The Changing Landscape of Search The explosive growth of electronic data over the past 20-30 years, combined with a proliferation of systems and sources, has made traditional search methods increasingly inadequate. As my colleague Rock de Vocht explains, “Search” has become more of a problem than a solution. We’ve seen the rise of keyword search tools and early content/document management systems. These were built on the premise that careful structuring of documents (via taxonomies) would enhance discoverability. However, this approach has its limitations, as the human brain doesn’t categorize information neatly in folders. The challenge now is to adapt this understanding to effective AI solutions. The Importance of Context The key element missing from traditional search is context. This is evident in examples of GenAI "hallucination," where the output might sound correct but is ultimately inaccurate. Early semantic search engines attempted to address context over 15 years ago, but rigid structures and taxonomies can hinder understanding when they are too simplistic. Today, indexing has shifted to a more relational, cloud-based structure, leading to the emergence of vector search. However, even vector search reveals limitations, prompting the development of semantic graph indexing, which prioritizes context over mere statistical mapping. Enter the Context Engine The new frontier is the Context Engine, designed to understand queries in relation to the user. It can access disparate information silos within an organization to deliver real insights, rather than just a list of possibly relevant documents—often referred to as the "Ten Blue Links" of early search engines. Currently, GenAI cannot achieve this alone, but when integrated with a robust Context Engine, the results are impressive. This combination utilizes a process called Retrieval Augmented Generation (RAG), allowing for direct answers to specific queries (e.g., “What is my remaining annual leave entitlement?”). Conclusion So, is search dead? Absolutely not! As Context Engines and GenAI evolve, they will likely converge, leading to new nomenclature and applications. While I agree that the search market feels overcrowded, the AI-powered search segment is just beginning to take off, and it’s exciting to see where it leads. SimSage is a Cambridge-UK based company specializing in contextual search, RAG, and maximizing the potential of GenAI! #enterprisesearch #workplacesearch #aisearch #contextengines #rag

🚀 LangChain's SelfQueryRetriever: A Sophisticated Tool for Contextual Search! 𝙄𝙢𝙖𝙜𝙞𝙣𝙚 𝙩𝙝𝙞𝙨: You’ve got a vast database of information, and instead of hardcoding filters or manually crafting queries, you let 𝘆𝗼𝘂𝗿 𝗾𝘂𝗲𝗿𝘆 speak for itself. That’s where LangChain’s SelfQueryRetriever shines. 🌟 With 𝗦𝗲𝗹𝗳𝗤𝘂𝗲𝗿𝘆𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗿, you can translate plain, natural language questions into structured filters, leveraging the power of LLMs to do the heavy lifting. Whether it's an SQL-like database, vector store, or any structured dataset, the retriever dynamically builds and applies conditions. LangChain uses a 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱𝗤𝘂𝗲𝗿𝘆 Object, which is then converted into database-specific query expressions. The query is filtered through the Visitor Class and formatted into the appropriate database-specific function, making the search experience smarter and more human-like. By leveraging this 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱𝗤𝘂𝗲𝗿𝘆 method, LangChain has created a database-independent format for obtaining logical outputs from LLMs. It then seamlessly translates these outputs into database-specific query expressions. 🎯 𝙒𝙝𝙮 𝙞𝙩’𝙨 𝙖 𝙜𝙖𝙢𝙚-𝙘𝙝𝙖𝙣𝙜𝙚𝙧: Dynamic Filters: No more manually setting rules; your queries are the rules! Flexible Use-Cases: Works seamlessly with text, metadata, or even hybrid datasets. Iterative Refinement: Need precision? Adjust the natural language query and let the model take care of the rest. 🌐 𝙍𝙚𝙖𝙡-𝙬𝙤𝙧𝙡𝙙 𝙀𝙭𝙖𝙢𝙥𝙡𝙚: Imagine this in customer support systems—querying a knowledge base for "recent issues related to battery life after the last update" or "articles tagged with AI and written post-2022." The context-aware filtering is pure magic. For those looking to dive deeper, there’s an excellent notebook by 𝗽𝗴𝗼𝗹𝗱𝗶𝗻𝗴 that breaks this process into meaningful steps. Worth a look! 𝗡𝗼𝘁𝗲𝗯𝗼𝗼𝗸: https://lnkd.in/dU9kepDp #LangChain #AI #SelfQueryRetriever #GenAI #LLM

Just shipped a deeeeeep dive on building a member retrieval system using semantic search 🤿 It won a build bounty competition at Build Club 🥰 If you just care about how the semantic retrieval with embeddings works, this post covers just that 💖 🔎 Semantic search is where the search engine knows what you’re looking for, even if you don’t say the exact words. Embeddings are numeric representations of words, phrases, documents etc that capture semantics, like grammatical info, conceptual properties and relationships between words and phrases like synonyms and associations etc. Imagine you have the following search query: 😸 Feline = [0.88, 0.08] And the following word embeddings: 🚂 Train = [0.1, 0.9…] 🐈 Cat = [0.9, 0.1…] 🐶 Dog = [0.08, 0.15…] 🦁 Lion = [0.85, 0.05…] Using a semantic similarity algorithm designed for measuring how closely embeddings are related (cosine similarity is the most typical algorithm), we would get back the following results: 🐈 Cat = [0.9, 0.1…] 🦁 Lion = [0.85, 0.05…] 🐶 Dog = [0.08, 0.15…] 🚂 Train = [0.1, 0.9…] Cat, and Lion are closer in semantic meaning to “Feline” than Dog or Train. This process of organising data based on how similar it is to the search query is called Ranking. Things to watch out for: 🙈 Semantic search will always return results even if there are no relevant results. So you need to implement a check that there are relevant results first. 🪂 You will also need to implement an ejection step so that if there are relevant results, only relevant results are returned. In the example above, we only want “Cat” and “Lion”, not “Dog” and “Train”. Otherwise your searchers can scroll right down to irrelevant results which can be confusing. 💌 Read the deep dive here: https://lnkd.in/g_hzcnZt #ai #genai #generativeai #rag #retrievalaugmentedgeneration #chatbot #embeddings #search #semanticsearch

Automatic Data Categorization with Optimally Spaced Semantic Seed Terms Main Process: User Input: Users enter their search queries. Document Retrieval: The system retrieves relevant documents from large databases (e.g., World Wide Web). Semantic Parsing: Documents are analyzed to identify semantic terms and groups. Keyword Ranking: Semantic keywords are ranked and optimally combined. Category Accumulation: Keywords and descriptors are integrated into final categories. Display Results: Categories are displayed to the user. The process loops until user satisfaction is achieved. Benefits: Improves the accuracy of search results. Creates meaningful and understandable categories. #DataCategorization #SemanticSearch #MachineLearning #AI #DataScience #BigData #SearchEngineOptimization #NaturalLanguageProcessing #TechInnovation