Anannya Das

Large Language Model Confidence Estimation via Black-Box Access Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh, Soham Dan, Prasanna Sattigeri June 2024. https://lnkd.in/dk5VTb2Z "Estimating uncertainty or confidence in the responses of a model can be significant in evaluating trust not only in the responses, but also in the model as a whole. In this paper, we explore the problem of estimating confidence for responses of large language models (LLMs) with simply black-box or query access to them. We propose a simple and extensible framework where, we engineer novel features and train a (interpretable) model (viz. logistic regression) on these features to estimate the confidence. We empirically demonstrate that our simple framework is effective in estimating confidence of flan-ul2, llama-13b and mistral-7b with it consistently outperforming existing black-box confidence estimation approaches on benchmark datasets such as TriviaQA, SQuAD, CoQA and Natural Questions by even over 10% (on AUROC) in some cases. Additionally, our interpretable approach provides insight into features that are predictive of confidence, leading to the interesting and useful discovery that our confidence models built for one LLM generalize zero-shot across others on a given dataset."

1

On AI Appreciation day, I shared my views with The Economic Times on how Artificial Intelligence, grounded in contextually-relevant information, is reshaping business dynamics. AI's ability to uncover real-time insights and democratize complex data is revolutionizing decision-making across all levels of organizations. However, to harness AI's full potential responsibly, businesses must invest in robust data integration and governance frameworks. Read more here: https://lnkd.in/gFAUbtaN

22

As enterprises grapple with AI strategy and emerging regulatory pressure, high-performing companies are establishing cross-functional AI councils to align AI use with enterprise strategy, values, and regulatory requirements. Recently one of our healthcare clients was approached to participate in a research partnership to build out a disease specific large language model - thanks to the establish risk management practices, they were able to ask the right questions and implement appropriate guardrails. In my most recent Forbes Technology Council article, I share: • Key roles to include on an AI council (hint: it's more than just tech experts) • Core responsibilities and deliverables • Real-world examples of successful AI councils • Common pitfalls to avoid and how to overcome them Whether you're just starting your AI journey or looking to level up your governance, AI councils are critical for staying competitive while mitigating risks. With incoming regulations like the EU AI Act, having a strong AI governance structure is no longer optional. Check out the full article below to learn how to build a successful AI council for your organization 👇 #AI #AIGovernance #ResponsibleAI #Innovation

12

6 Comments

Have you ever thought of designing a Advanced search solution using combined power of Graph DB And Opensearch vector Engine to address complex use cases like Evidence Catalog search in healthcare domain ??? 📣📣📣 This new Blog post jointly from Amazon Web Services (AWS) and ZS is now published to address this !!! Our joint collaborative Solution design effort with ZS showcases advanced Enterprise Graph Retrieval-Augmented Generation (RAG) solution, utilizing Amazon OpenSearch Service, Amazon Neptune, and cutting-edge unstructured data processing powered by state-of-the-art Generative AI. 💡 The solution facilitates efficient search and retrieval of a vast array of documents, encompassing study protocols, research papers, clinical guidelines, and beyond. This platform aids pivotal decision-making processes, such as strategy formulation and planning for evidence generation, by empowering users to swiftly locate specific, pertinent information from both structured data sources (like organized databases) and unstructured data sources (like written reports). Kudos to ZS Team for this breakthrough solution with Graph RAG at scale on Amazon Web Services (AWS)!!! Blog Link : https://lnkd.in/gRXhYsnt Shivam Shivam Sandeep Verma Alex Turok Gourang Harhare, Kevin Phillips Jon Handler Ninad Phatak Guru Bala Sainath Bandhakavi Shashidhar Boga Praveen Poddar #LLM #GenerativeAI #GraphDB #Opensearch #AWS #Data #vectorDB #RAG #Innovation #Healthcare #EnterpriseGraph #SemanticSearch

32

1 Comment

I am extremely bullish on the growth of AI – and SingleStore – in India. I sat down with Amit Raja Naik of Analytics India Magazine (AIM) to discuss why we’re so excited about the Indian market, why simplicity is key to scaling AI, and why the ultimate use of data is to improve the lives of the underserved. Check out my full interview here: https://lnkd.in/eD7zKj8h #AIM #AI #Data #SingleStore

98

2 Comments

A crisp and enriching framework by McKinsey & Company to help us all understand #genAI #adoption strategies better. With #humans and #humanity as the anchor of technological advancements using #GenAI, we are in for an exciting and promising journey ahead. #IBelongHere #AIinnovation

5

1 Comment

My Journey in AI & Blockchain From startups to large MNCs, and now as Director of AI at Foundry, my journey has been anything but traditional. What began as a curiosity in machine learning has evolved into a passion for decentralized technologies like AI and Blockchain—tools that I believe will transform industries beyond our imagination. I’ve had the privilege of building AI solutions from scratch, and now, my team at Foundry is empowering institutions with tools to mine and stake digital assets. But here’s the lesson I’ve learned: technology is only as powerful as the problems it solves. 🔑 AI and Blockchain are not just buzzwords—they are the foundation of a more secure, efficient, and transparent future. The next decade will be about decentralizing power, reducing friction in industries like finance, healthcare, and supply chain, and creating fairer systems for everyone. Let’s build that future together. 💡 #AI #Blockchain #Innovation #DecentralizedFuture

60

1 Comment

While the consumers like you & me are lured into the speed, efficacy and magic of #GenAI, the worried #AI developers are looking for ways to understand the inner working of LLMs for future safety. While #interpretability is one way to address it (Anthropic showcased it earlier), #explainability, especially by #LLM itself (since they are good at producing prose or poetry), could be another. The latest paper published a week ago explains #SaySelf, a training framework that attempts to addresses this requirement by improving LLM confidence estimates. It uses multiple sampled reasoning chains, semantic clustering and #RL with a newly crafted custom-reward function to calibrate confidence estimates and finetune an LLM. SaySelf reduces #confidence #calibration error by 30%. Link to the paper & code in comments.

17

1 Comment

Great article covering PII in ML models and how to keep your data private. #ai #langchain

6

I'm thrilled to announce my participation as a speaker at the highly anticipated 3AI #SPECTRA event, a confluence of cutting-edge AI and GenAI discussions with over 150+ renowned speakers across 3 insightful days. 🌟 During my session, I'll be delving into the best practices to maximize ROI for Generative AI initiatives within enterprises. Drawing from my hands-on experience driving multiple large-scale GenAI projects, I'll share valuable insights and strategies to unlock the true potential of this transformative technology. 🚀 Get ready to uncover: ✅ Key considerations for successful GenAI adoption and implementation at an enterprise scale. ✅ Proven methodologies to measure and optimize GenAI's impact on business outcomes. ✅ Real-world case studies and lessons learned from driving GenAI initiatives across diverse industries. ✅ Actionable tips to navigate challenges and accelerate time-to-value for your GenAI investments. Whether you're an AI leader, technology enthusiast, or a business decision-maker exploring the power of GenAI, this session promises to be an invaluable opportunity to gain practical knowledge and stay ahead of the curve. 🔍 Don't miss this chance to witness the live event and engage with industry experts pushing the boundaries of AI and GenAI. Join me at 3AI SPECTRA, where we'll collectively shape the future of intelligent systems and drive transformative change. 🌐 Mark your calendars and secure your spot today! Let's unlock the full potential of GenAI together. 💪 #3AISPECTRA #GenerativeAI #EnterpriseAI #AIAdoption #ROIOptimization #TechnologyLeadership

30

The Analytics India Magazine has a piece on an Indian startup "building a private Perplexity for enterprise" https://lnkd.in/ec-_xt8x. #generativeai #perplexity #startup #artificialingelligence #startup #perplexity #analyticsindiamagazine

5

Short and very knowledgeable course from DeepLearning gives a high-level view on dense retrievals, rerank, num generation, and many more concepts around use cases for LLMs. https://lnkd.in/ex9sZrbV"

7

2 Comments

Databricks surely gonna be ruling the Data and AI space. A right step towards right direction at right time 💥 Databricks has acquired Tabular, a data management startup founded by Apache Iceberg creators, for over $1 billion. The acquisition aims to build a unified data management platform based on open lakehouse formats, reducing silos and friction for customers. This move highlights Databricks' commitment to open formats and open source data in the cloud. Read the detailed news over here: https://lnkd.in/gqVcxjcv #dataengineering #dataplatform #datastrategy #storage #nosql #cloud

6

Ola Krutrim , India’s first AI unicorn, recently launched its LLM, Krutrim-7B-chat. This new model is trained in 10 Indian languages and is available on Databricks Marketplace. It can generate text in Hindi, English, Tamil, Telugu, Marathi, Malayalam, Gujarati, Bengali, and Kannada. This is the first of several India-specific models trained by Ola Krutrim. It has been trained on a massive dataset of Indic languages— “Krutrim delivers text infused with Indian sensibilities and cultural awareness.” Check out the model here. The announcement came to light at the Data Intelligence Day in Mumbai on May 10, 2024, by Gautam Bhargava, VP & Head of AI Engineering at Krutrim. “We’re excited to be at the forefront of building foundational LLMs trained on Indian languages to better serve our customers in India. We have been working very closely with the Databricks team to pre-train and finetune our foundational LLM,” said Bhargava. He added, “We are confident that Krutrim, our base model, will be able to provide high-quality responses in Indian languages and dialects.” According to Bhargava, Krutrim was trained using Databricks Mosaic AI Training’s GPU orchestration and system optimisations, enabling state-of-the-art generative AI solutions for customers globally. The model is now available on the Databricks Marketplace.

49

1 Comment

If you are still thinking that AI is just a hype, you should listen the conversation between Mukesh Ambani and NVIDIA CEO Jensen Huang. Why ? Mukesh Ambani bullishly talks about Generative AI models, GPUs, need of open source models etc. He takes up the challenge to surprise world with AI capabilities in coming years. He is not someone who has been leading IT/Software companies day in day out like Narayan Murthy/ Shiv Nadar/Bill Gates etc. What it means? AI has already arrived and every big firm, irrespective of their traditional business, wants to build business and create impact using this. AI is not just another cool tech anymore. It is the industrial revolution changing every facet of our life. Embrace it or get ready for classic "Kodak Case study" in your career and business. Watch here: https://lnkd.in/ejvnCMHZ

10

Clean data is very crucial for any AI/ML product to succeed. Sounds like Same Old Gyan and more than a decade old problem. More often than not Source DQ, Master Data Management, Data Integrity in a large scale enterprise are top problems that can impact quality of Pred/Gen AI solutions !!! Cutting edge data "foundation" is an undeniable key to the success of bleeding edge AI :) 2 hours of video processing ability may not be the ultimatum to solve this classic :p PC: LLAMA3.1 #ai #genai #datafoundation

21

3 Comments

!!AI Adoption!! In a recent study, over 40% of Japanese firms revealed they lack AI adoption plans, highlighting a significant challenge in harnessing the full potential of artificial intelligence. I wonder how is Croatia doing comparatively? However, both countries face similar obstacles: workforce readiness and integration of AI in traditional industries. Embracing AI can drive innovation and economic growth, making it imperative for businesses worldwide to develop comprehensive AI strategies. #AI #Innovation #BusinessStrategy #Croatia #Japan #FutureOfWork https://lnkd.in/dXSjP5hJ

27

3 Comments

Every organization’s Data + AI journey begins with fundamental questions like: - Is data centralization truly necessary for AI enablement across multi-siloed data? - Can Snowflake, Databricks, and other providers' multi-catalog and AI agent systems offer flexible options for compute, storage, knowledge, and models without requiring centralized data? Why is this so critical? Ans: Because every company seeks to harness AI capabilities to quickly enable and accelerate experimentation, driving growth and innovation while minimising costs and time investment. Discover how Dview’s Dsense platform seamlessly integrates diverse data sources, enabling effortless insights without the need for traditional data centralization from the start. Feel free to schedule a detailed discussion using my Calendly link: [ https://calendly.com/kauts ]. #AI #DataAnalytics #DataIntegration #AIEnablement #Dsense

36