Ashish Vaswani

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

6

1 Comment

#LLM sizes will be a significant consideration in the next generation of products that we will be witnessing in the coming years. Router models cannot be the go-to solution eventually. As a founder in this domain, one needs to think beyond constrained boxes and democratize these powerful systems as much as possible for small and medium business. We are planning to set things straight in the coming months. #Angel #angelfunding #VC #investments #VCfunding #AI #AIStartup #Quantization #LLMs #LLM

4

The April 25th seminar was viewed by over 16,000 people regarding 'Meta Llama 3 Fine tuning, RAG, and Prompt Engineering for Drug Discovery' to learn about the basics of advanced LLMs as they apply to a scientific application. https://lnkd.in/gy-FH6aj This week, the use of LLM agents to further automate and improve the performance of Generative AI was explored. In specific, an ACS publication used Ontological Knowledge Graphs and multiagent strategies with Llama-2 models to improve materials design. An innovative CrowChem AI Agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore better than using GPT-4 alone. Video discussion: https://lnkd.in/gMdWHf45 “Given a complex task like writing software, a multi-agent approach would break down the task into subtasks to be executed by different roles - such as a software engineer, product manager, designer, QA (quality assurance) engineer, and so on - and have different agents accomplish different subtasks.”  - Andrew Ng, Founder of Deeplearning.ai, April 17, 2024.

84

3 Comments

Large language models (LLMs) are increasingly being explored for their potential in medicine, but applying them effectively to complex clinical scenarios remains a challenge. The paper "MDAgents: an adaptive collaboration of LLMs for medical decision-making" by Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park (NeurIPS 2024) introduces a novel framework aimed at addressing this gap. MDAgents structures collaboration among LLMs dynamically, assigning roles—either independent or group-based—based on the complexity of the medical task. This design mimics real-world clinical decision-making processes, where collaboration is tailored to the case at hand. Evaluated on 10 medical benchmarks, MDAgents achieved leading performance on 7 tasks and demonstrated an 11.8% accuracy boost when incorporating external medical knowledge and moderator reviews. A case study further illustrates how the framework synthesizes differing perspectives among LLMs to reach accurate, consensus-driven diagnoses. This framework stands out for its adaptability and its emphasis on collaboration. Ablation studies underscore the importance of the system’s components, from its ability to classify medical complexity to the integration of multi-modal reasoning. By reflecting the dynamic, consultative nature of clinical decision-making, MDAgents provides a thoughtful approach to enhancing LLM-assisted medical diagnosis. To me, this paper is interesting because it takes a practical step toward aligning AI capabilities with the intricacies of healthcare. By emulating real-world medical collaboration, MDAgents not only improves performance but also lays the groundwork for more robust and trustworthy AI systems in clinical settings. What are your thoughts on how AI frameworks can better reflect real-world decision-making processes in fields like healthcare? paper: https://lnkd.in/gd5fX4Ef #artificialintelligence #AIhealthcare #LLMs #AIAgents

11

📜 A new blog post explaining the fundamentals of RL for LLMs—the perfect place to intuitively understand PPO and RLHF from first principles. 👇👇 Pre-trained LLMs are often compared to Lovecraft's Shoggoth, a monstrous, shape-shifting beast. PPO is the algorithm that brought it under control. Take a look! 🔗 https://lnkd.in/eXqDvrUV

29

1 Comment

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

22

1 Comment

Falkonry (Suhas Mehta and Ashis Khan) and iba AG (Eric Snyder) will present how steel makers are pivoting to AI & Data driven decision support system at AISTech 2024 today (May 6, 2024) - please join us if you are interested. I co-authored this paper with Eric Snyder of iba AG.

65

5 Comments

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

Over the past few months at Assembled, we've been exploring how Large Language Models (LLMs) can enhance our engineering workflows. One area where they've made a significant impact is in writing comprehensive tests. Traditionally, writing thorough tests can be time consuming and sometimes gets sidelined due to tight deadlines. By leveraging LLMs, we've been able to generate robust tests much more quickly—tasks that used to take hours now take just minutes. This shift has brought: - Increased adoption of testing: Engineers who previously wrote fewer tests are now consistently incorporating them into their workflow. - Time savings: We've collectively saved hundreds of hours, allowing us to focus more on developing new features and refining existing ones. - Enhanced code quality: Our codebase is better tested, giving us greater confidence in our deployments. I've shared more about our approach, including examples and lessons learned, in a recent blog post. 👉 Read more here: https://lnkd.in/gKw3djQu We're excited about the progress we're making and the insights we've gained. If you're interested in the intersection of LLMs and software engineering, or if you're passionate about building tools that transform customer support, I'd love to connect. P.S. We're hiring! Check out our open roles: https://lnkd.in/gSr_wqV7

74

3 Comments

How do you make large-scale knowledge graphs actually usable for AI retrieval? We’ve been tackling this question with memory fragment projection using cognee. The idea is simple: 💡 Instead of analyzing the entire graph, we extract focused subgraphs (memory fragments) that are small enough for in-memory exploration but rich enough to support AI queries. In the blog, Hajdu László, Ph.D. walks us through: 🚶‍➡️ Subgraph filtering methods (community-based, neighborhood-based, etc.) and their tradeoffs 🚶‍♀️‍➡️ How cognee integrates with Neo4j to detect communities and project graph fragments 🚶‍♂️‍➡️ A practical example of projecting memory fragments from Neo4j’s knowledge graph This approach lets us: ✅ Optimize AI retrieval pipelines with GraphRAG ✅ Maintain flexibility while focusing on relevant graph layers ✅ Scale down the complexity without losing context If you’ve worked with knowledge graphs or retrieval pipelines, you know the pain of balancing scale and precision. This solution makes that process much more approachable. Here’s the blog 👇 https://lnkd.in/d5js2psf Would love to hear your feedback or challenges you’ve faced in this space!

21

Read our latest published research paper about SuperCoder

27

Are LLMs better reward models? Recent alignment research has seen the convergence of two approaches: reward modeling and LLM-based evaluation. Traditional reward models, used for Reinforcement Learning from Human Feedback (RLHF), are being augmented with LLMs' reasoning and critique abilities to provide more nuanced evaluations. Henry Broomfield writes about key developments in the literature: from early experiments with Reinforcement Learning from AI Feedback (RLAIF) to the latest model architectures that can generate their own critiques during evaluation. While open questions remain, we’re excited to see how these developments influence how we train and align the next generation of language models. Here’s the full article: https://lnkd.in/ewmQ8FYy

24

SPACE2: Improved computational epitope profiling using structural models identifies a broader diversity of antibodies that bind to the same epitope. Web server: https://lnkd.in/gJT3wgh8 Paper: https://lnkd.in/gwEs93Fq Group your antibodies by what epitope they bind to on Tamarind Bio! Abstract: The function of an antibody is intrinsically linked to the epitope it engages. Clonal clustering methods, based on sequence identity, are commonly used to group antibodies that will bind to the same epitope. However, such methods neglect the fact that antibodies with highly diverse sequences can exhibit similar binding site geometries and engage common epitopes. In a previous study, we described SPACE1, a method that structurally clustered antibodies in order to predict their epitopes. This methodology was limited by the inaccuracies and incomplete coverage of template-based modeling. In addition, it was only benchmarked at the level of domain-consistency on one virus class. Here, we present SPACE2, which uses the latest machine learning-based structure prediction technology combined with a novel clustering protocol, and benchmark it on binding data that have epitope-level resolution. On six diverse sets of antigen-specific antibodies, we demonstrate that SPACE2 accurately clusters antibodies that engage common epitopes.

32

🧠 Exciting Frontiers in Bio-Inspired AI! 🤖 My advisor Prof. Alexander Ororbia II just had an incredible conversation with Machine Learning Street Talk (MLST), one of my favorite podcasts of all-time! In this episode, they dive deep into mortal computation and neurobiologically-plausible learning algorithms. Key highlights: • Mortal Computation: A fresh perspective on biomimetic intelligence • Predictive Coding: Unlocking efficient and adaptable AI systems • Forward-Only Learning: Challenging traditional backpropagation • Neural Generative Coding: Bridging the gap between AI and biological intelligence Prof. Ororbia's insights on implementing these bio-inspired approaches on neuromorphic hardware are truly groundbreaking. His vision for AI systems that closely align with biological intelligence is both inspiring and thought-provoking. At our lab, The NAC Lab, we're excited to be at the forefront of this revolution in AI. We're constantly exploring new ways to integrate these bio-inspired principles into our research and development. Are you passionate about pushing the boundaries of AI? We'd love to hear your thoughts! #BioInspiredAI #MortalComputation #NeuromorphicComputing #AIResearch #MachineLearning #ArtificialIntelligence #PredictiveCoding #ActiveInference [1] https://lnkd.in/gf5GWRMh

60

1 Comment

Nice demo talk from the creator of #Devin during the recent AI Engineer #worldfair. Nice demonstration and articulation of what the "autonomous software engineer" agent can do. Given scrutiny about previous Devin demos, I want to assume these were all bullet proof authentic. There were Q&A, and I wish someone had asked for his opinion on allegations for fake demos, and comparison to open Devin and other open source projects. Anyhow, IMHO, the key point he made - yyX more (human) software developers that are zzX more efficient - basically by lowering the bar and delegation. What he didn't say: 1) These engineers will also be much cheaper 2) There will also be no code / low code versions for non engineers. These raise interesting questions about the future of the software business. As engineering becomes cheaper and more accessible, will we see a shift in build vs buy, especially in the application layer - assuming hyper customized experiences could be tailor made at low cost and low risk. https://lnkd.in/e4KH4CfC #ai #innovation #productivity #startups #futureofwork

2

AI models trained solely on public data are often not sufficient for protein design tasks. Experimental data might help! 𝗣𝗿𝗼𝗽𝗲𝗿𝘁𝘆 𝘀𝗰𝗼𝗿𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗲𝗱 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 Language models such as ESM, Ankh, and ProstT5 trained on the available universe of protein sequence (sometimes including structural information as well) have been shown to perform well at predicting properties blindly. But well, they aren't trained on your sequences. Fine-tuning incorporates internal experimental data to existing models, creating predictors with increasing performance on your specific sequences. In “Fine-tuning protein language models boosts predictions across diverse tasks”, the authors demonstrate that these specialized models outperform generalized ones for many “mutational landscapes.” These include predicting stability, binding affinity, and fluorescence. See the attached graphic I made for their accuracies on large public benchmarks and how much improvement is observed with fine-tuning. Easily fine-tune a model on your data on Tamarind: https://lnkd.in/gb57jpS6 𝗔𝗰𝘁𝗶𝘃𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴-𝗔𝘀𝘀𝗶𝘀𝘁𝗲𝗱 𝗗𝗶𝗿𝗲𝗰𝘁𝗲𝗱 𝗘𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻 Another approach is to use experimental data as the only source of guidance. With this, we can train completely new models from a spreadsheet containing sequences and the associated properties. With each round of testing, this protocol trains a more informed scoring function, recommending mutations, and learning from the experimental results of these mutations. The authors apply ALDE to an optimization problem, of five residues in the active site of an enzyme. In three rounds of wet-lab experimentation, they improve yield from 12% to 93%. It's valuable to note here that there is no enzyme-specific methodologies applied here. It's simply producing a computational predictor for a property of interest, given mutations. This means it should be a feasible, and likely more efficient alternative to directed evolution in therapeutic contexts as well. ALDE on Tamarind: https://lnkd.in/gEQfMk8x Get in touch if you'd like to try these methods with your internal data on Tamarind Bio, in our secure confidential cloud!

40

🚨 Big News from Meta Connect! 🚨 Meta just dropped its Llama 3.2 models on Watsonx, including the first multimodal 11B and 90B Vision models! Key Highlights: - Image-in, text-out support for tasks like document understanding and image captioning. - High-res image support (up to 1120x1120) for classification, OCR, and more. - Lightweight Llama 3.2 models (1B & 3B) run on most devices with low latency and enhanced privacy. - New multimodal safety model: Llama Guard. Meta continues leading with efficient, open-source AI models! Groq now support Llama 3 models for even faster AI Performance #AI #Meta #Llama32 #TechInnovation #MultimodalAI #Watsonx #genai

26

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

I am excited to share that Dr. Feifang Hu, my co-founder and PhD advisor, will present at the Regulatory-Industry Statistics Workshop (RISW) on Sept 27, 2024. He will discuss practical considerations of covariate adjusted inferences in various classic and innovative designs. Join us to explore how statistical innovations and AI are revolutionizing clinical drug development! #RISW2024 #ClinicalTrials #CovariateAdjustment #AI

20