Krasty Soft’s Post

What is semantic search? Semantic search is a modern technique for search engines like Elastic to understand what you mean when you type something. Instead of just finding documents with the exact words you type, it tries to find documents that match the meaning behind your words. It uses technology like artificial intelligence to do this. Here's how it works: First, it turns text into numeric vectors representing your request. Then, it compares these vectors to other numbers that represent documents. It looks for the most similar ones and gives you those as results. "Context" is important in semantic search. That means where you are, what you've searched for before, and even the words around what you're looking for. For instance, if you type "slacks", it knows if you mean pants or trousers. Another important thing is understanding what you want. Semantic search tries to figure out if you're looking for information, trying to buy something, or something else. It then shows you the most fitting results based on what you need. The benefits of semantic search for business are obvious. Semantic search helps understand what customers want, whether it's information, buying something, or just exploring. This understanding can lead to more sales and happier customers, building a better relationship between customers and the brand. Feel free to reach out if you want to discuss how semantic search can benefit your business! #elasticsearch #elastic #search #ai #semanticsearch #krastysoft

Just shipped a deeeeeep dive on building a member retrieval system using semantic search 🤿 It won a build bounty competition at Build Club 🥰 If you just care about how the semantic retrieval with embeddings works, this post covers just that 💖 🔎 Semantic search is where the search engine knows what you’re looking for, even if you don’t say the exact words. Embeddings are numeric representations of words, phrases, documents etc that capture semantics, like grammatical info, conceptual properties and relationships between words and phrases like synonyms and associations etc. Imagine you have the following search query: 😸 Feline = [0.88, 0.08] And the following word embeddings: 🚂 Train = [0.1, 0.9…] 🐈 Cat = [0.9, 0.1…] 🐶 Dog = [0.08, 0.15…] 🦁 Lion = [0.85, 0.05…] Using a semantic similarity algorithm designed for measuring how closely embeddings are related (cosine similarity is the most typical algorithm), we would get back the following results: 🐈 Cat = [0.9, 0.1…] 🦁 Lion = [0.85, 0.05…] 🐶 Dog = [0.08, 0.15…] 🚂 Train = [0.1, 0.9…] Cat, and Lion are closer in semantic meaning to “Feline” than Dog or Train. This process of organising data based on how similar it is to the search query is called Ranking. Things to watch out for: 🙈 Semantic search will always return results even if there are no relevant results. So you need to implement a check that there are relevant results first. 🪂 You will also need to implement an ejection step so that if there are relevant results, only relevant results are returned. In the example above, we only want “Cat” and “Lion”, not “Dog” and “Train”. Otherwise your searchers can scroll right down to irrelevant results which can be confusing. 💌 Read the deep dive here: https://lnkd.in/g_hzcnZt #ai #genai #generativeai #rag #retrievalaugmentedgeneration #chatbot #embeddings #search #semanticsearch

"Ultimately, the end user isn’t concerned about the intricacies of whether it’s a vector search, keyword search, rule-driven search or even a “phone a friend” search. What matters most to the user is getting the right answer. Rarely does this come from relying solely on one methodology. Understand your use case and validate your test scenarios … and… don’t be lured by shiny objects just because they’re popular." https://lnkd.in/e3TxBs8M #heroku #salesforceai #ai

Sometimes searching for a string in large documents and extensive text can be very challenging, especially when time is limited. As a data scientist, I face this issue because, in the world of data, every document is important for gaining valuable insights. Simple searches can be slow, and even regex can take a long time with millions of records. Enter FlashText - a game-changer! It’s 82 times faster than regex and perfect for efficient string searching. 🚀 FlashText is an algorithm based on the Tire dictionary data structure and inspired by the Aho-Corasick Algorithm. The Trie dictionary is a set of predefined keywords and phrases used for fast and efficient keyword extraction and replacement within text. The way it works is, first, it takes all relevant keywords as input. Using these keywords, a Trie dictionary is built. The FlashText algorithm has three major parts: 1. Building the Trie dictionary 2. Searching for keywords 3. Replacing keywords Why FlashText? ⚡ Speed: Unlike traditional methods like regular expressions, FlashText can search and replace keywords in text with lightning-fast speed, even for large volumes of data. 💡 Efficiency: FlashText processes text in O(n) time complexity, making it highly efficient for applications requiring quick keyword extraction or replacement. 👍 Ease of Use: With a simple API, FlashText is easy to integrate into your existing projects. Define your keywords, and FlashText handles the rest! Here! you can read more about this https://lnkd.in/gG4YtMB7 #DataScience #FlashText #Algorithm #Efficiency #BigData #MachineLearning #AI #TechInnovation

Generative AI & Traditional Search Engines Working Together The future of search is more than just finding information—it’s about understanding context, providing insights, and delivering results that make sense. With Generative AI and powerful search engines like Elasticsearch, Solr, and OpenSearch, we have the technology to take search experiences to new heights. Here’s how each tool brings unique strengths to the table: Generative AI is transforming search by understanding context and generating responses in natural language. It’s ideal for answering complex questions, summarizing information, and offering a more conversational search experience. Meanwhile, traditional search engines excel in areas that require speed, precision, and real-time data handling. Here’s how they stack up: - Elasticsearch: Known for its lightning-fast retrieval and real-time indexing, Elasticsearch is a top choice for environments where performance and scale matter, such as monitoring systems and e-commerce.    - Solr: As an Apache project, Solr provides highly customizable indexing and query capabilities. It’s a great option for complex data structures and offers strong performance in environments where exact matching and custom queries are needed.    - OpenSearch: Originally derived from Elasticsearch, OpenSearch maintains a focus on scalability and flexibility, making it a solid choice for applications needing open-source, highly customizable search options. Combined Power: Hybrid Search Solutions These tools, combined with Generative AI, offer a comprehensive solution for both structured and unstructured data needs. Imagine Elasticsearch or Solr retrieving relevant data, with Generative AI shaping those results into meaningful insights. This hybrid approach allows us to leverage the speed and precision of search engines alongside the context and conversational ability of AI. The result? A search experience that’s fast, insightful, and adaptive to user needs. It’s not about choosing one tool over another—it’s about integrating these technologies to deliver search that’s truly impactful. I’d love to hear others' thoughts on this approach and learn about any research that explores the potential of combining these tools. What do you think? #SearchTechnology #GenerativeAI #Elasticsearch #Solr #OpenSearch #Innovation #BigData #ESRE

#LLM can help you simplify using #data to get simple and usable analytics. In this article, Daniel Ong demonstrates how to build a great Semantic search using Snowflake Cortex on top of a dataset of Bright Data, containing #webdata from the AirBnB website. Example of using #webdata for the best #ai and #analytics, leveraging the Snowflake-BrightData partnership. Michael Rhodes Félix Salacroup Judit Daniel Pascal Ferro Tony Young https://lnkd.in/dw_c4mxa

🚀 LangChain's SelfQueryRetriever: A Sophisticated Tool for Contextual Search! 𝙄𝙢𝙖𝙜𝙞𝙣𝙚 𝙩𝙝𝙞𝙨: You’ve got a vast database of information, and instead of hardcoding filters or manually crafting queries, you let 𝘆𝗼𝘂𝗿 𝗾𝘂𝗲𝗿𝘆 speak for itself. That’s where LangChain’s SelfQueryRetriever shines. 🌟 With 𝗦𝗲𝗹𝗳𝗤𝘂𝗲𝗿𝘆𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗿, you can translate plain, natural language questions into structured filters, leveraging the power of LLMs to do the heavy lifting. Whether it's an SQL-like database, vector store, or any structured dataset, the retriever dynamically builds and applies conditions. LangChain uses a 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱𝗤𝘂𝗲𝗿𝘆 Object, which is then converted into database-specific query expressions. The query is filtered through the Visitor Class and formatted into the appropriate database-specific function, making the search experience smarter and more human-like. By leveraging this 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱𝗤𝘂𝗲𝗿𝘆 method, LangChain has created a database-independent format for obtaining logical outputs from LLMs. It then seamlessly translates these outputs into database-specific query expressions. 🎯 𝙒𝙝𝙮 𝙞𝙩’𝙨 𝙖 𝙜𝙖𝙢𝙚-𝙘𝙝𝙖𝙣𝙜𝙚𝙧: Dynamic Filters: No more manually setting rules; your queries are the rules! Flexible Use-Cases: Works seamlessly with text, metadata, or even hybrid datasets. Iterative Refinement: Need precision? Adjust the natural language query and let the model take care of the rest. 🌐 𝙍𝙚𝙖𝙡-𝙬𝙤𝙧𝙡𝙙 𝙀𝙭𝙖𝙢𝙥𝙡𝙚: Imagine this in customer support systems—querying a knowledge base for "recent issues related to battery life after the last update" or "articles tagged with AI and written post-2022." The context-aware filtering is pure magic. For those looking to dive deeper, there’s an excellent notebook by 𝗽𝗴𝗼𝗹𝗱𝗶𝗻𝗴 that breaks this process into meaningful steps. Worth a look! 𝗡𝗼𝘁𝗲𝗯𝗼𝗼𝗸: https://lnkd.in/dU9kepDp #LangChain #AI #SelfQueryRetriever #GenAI #LLM

We're halfway through 2024, and no AGI yet! But it looks like we might skip AGI and go straight to SSI 😉 Until the next AI buzzword emerges, here's what really matters. Two main concepts drive the form factor of LLM applications: 1️⃣ Retrieval Augmented Generation (RAG)  2️⃣ Agentic Systems Combine the two concepts driving AI Applications today, and you get: 🥁🥁🥁🥁🥁🥁🥁🥁 🌟Agentic RAG🌟 It combines RAG's information retrieval paradigm with an AI agent's ability to use tools, reason, and plan. You can think of it as a 2-for-1 bundle deal on building AI applications. To understand Agentic RAG,Imagine an AI that can: 1️⃣ Decompose complex tasks 2️⃣ Choose the right tools for each sub-task 3️⃣ Decide when to retrieve information vs. use its own knowledge 4️⃣ Execute multi-step problem-solving strategies Now you can stop imaging, as we've just released an in-depth guide on implementing Agentic RAG using: ↦ Anthropic's Claude 3.5 Sonnet for the Agent's brain 🧠 ↦ LlamaIndex for building the Agent integration 👷🏾♀️ ↦ MongoDB as the memory provider 💽 Here's what you'll learn: ✅ Build a full Agentic RAG pipeline ✅ Optimize embedding processes ✅ Create AI agents with advanced reasoning skills ✅ Leverage MongoDB for efficient knowledge retrieval Tutorial: https://lnkd.in/ehgrUycR Code: https://lnkd.in/gR3bqyjx #artificialintelligence #genai #ai #agenticsystems #aiagents #mongodb #claude

I'm thrilled to share my latest blog: Elevate Your Search Accuracy with Vector Search in Vertex AI! Discover how Vector Search, developed by Google Research, revolutionizes information retrieval by going beyond keyword limitations. Learn about the advantages of multi-modal search, which integrates text, images, audio, and video to provide more accurate and relevant results. Explore how embeddings transform diverse data types into meaningful mathematical representations, enhancing search experiences across various applications. Unlock the potential of semantic search with Vertex AI and improve your search accuracy today! Check out my blog👇 https://lnkd.in/ep2P6g-a #AI #MachineLearning #DataScience #VertexAI #VectorSearch #GoogleCloud #InformationRetrieval #TechInnovation #MultiModalSearch

My GenAi GoTo stack (currently) 🤗 In the really fast evolving landscape of AI and machine learning, choosing the right stack is important. After evaluating various options, I settled on a combination that offers robust performance, flexibility and scalability. Here's why this is currently my universal stack of choice: 🔍 Model Provider: Llama 3.1 by Meta The Llama 3.1 model by Meta continues to impress with its performance and versatility, making it my preferred choice for natural language processing tasks. Whether it’s for research or deployment, Llama 3.1 provides the power and efficiency needed to drive impactful results. 🛠 LLM Frameworks: LlamaIndex LlamaIndex offers a streamlined framework to manage and fine-tune large language models efficiently. Its intuitive interface and powerful capabilities make it a go-to for developing custom AI applications. 💾 Databases: Postgres When it comes to reliable and scalable database solutions, Postgres is unbeatable. Its robustness and rich feature set make it ideal for managing complex datasets in AI and machine learning applications. Plus it is the best Vector-capable SQL database 😜 📊 Monitoring: Weights & Biases For model tracking, experiment management, and performance visualization, Weights & Biases stands out. It seamlessly integrates into my workflow, ensuring that every model iteration is well-documented and optimized. 🚀 Deployment: Hugging Face Deploying AI models has never been easier, thanks to Hugging Face. With its expansive model hub and easy-to-use APIs, Hugging Face enables rapid deployment, scaling, and sharing of state-of-the-art models ... and I really like underdogs 💪 🌱 Why This Stack? This combination balances cutting-edge technology with practical, real-world usability. It empowers me to build, monitor, and deploy AI solutions faster and more effectively . Whether we are working on large-scale enterprise solutions or innovative research projects, this stack is built to adapt and scale with our needs. What does your AI/ML stack look like? I’d love to hear your thoughts! I will also be posting all the details for each individual layer of this stack, so if you are interested, please follow me on Linkedin. Please ♻️ repost this if you think it makes sense for you! #AI #MachineLearning #TechStack #Llama3 #Postgres #WeightsAndBiases #HuggingFace #LLM #DataScience