Rush Bartlett

As the discourse shifts from models to compound AI systems / agents, we need better AI benchmarks to evaluate multi-modal and multi-step task performance, especially in healthcare and life sciences. When we wrote the first paper demonstrating ChatGPT's performance on the USMLE, we chose the US Medical Licensing Exam as a benchmark for accessibility, speed, and ease. This benchmark was never intended to represent AI model performance on real-world clinical tasks. Today, I still see so many research teams and startups using benchmarks (like the USMLE) that are ill-fitted for assessing the true clinical or scientific performance and utility of the models they are developing for real-world contexts. Benchmark development may be seen as a "less sexy" area of research, but it is of paramount importance. Years after the rise of the transformer, we still lack adequate benchmarks for so many single-step tasks in biomedicine. With compound AI systems (i.e., architectures that integrate multiple AI models to perform complex tasks) emerging, we need new benchmarks for agentic behaviors. I'd argue that developing an agent with novel capabilities without at least proposing a companion benchmark (if an industry standard does not yet exist) may hinder the adoption of said agent, especially for high-stakes workflows. Designing more benchmarks that capture/simulate real-world clinical and scientific workflows will help us mitigate the major discrepancies observed between in silico and in vivo performance and better support safe + effective deployment of AI in biomedicine. There are already brilliant people focused here, and we need more. DMs are open if you're researching or working in this area of multi-step/multi-modal benchmarking in healthcare and life sciences! #healthcare #ai #artificialintelligence #generativeai

100

18 Comments

There appears to be a "Superstar Effect" in healthcare and biomedical AI research right now. A small percentage of publications are truly novel methods contributions that introduce new capabilities or behaviors; whereas a majority of contemporary research publications are focused on implementation science to assess the real-world utility and performance of new methods. This dynamic seems different from prior eras, when the translational infrastructure for implementing novel AI methods was largely unavailable. While methods research often captures more attention and accolades, both types of research are critical – novel methods without implementation science lacks translatability; implementation science without novel methods lacks novelty. If our goal is to invent and deploy transformative AI methods in real-world healthcare and life sciences contexts, we ought to continue encouraging and rewarding stellar implementation science work alongside breakthrough methods development.

90

3 Comments

"Healthcare innovation is accelerating at an unprecedented pace, and cardiology is on the brink of transformative change using new forms of remote monitoring and artificial intelligence (AI). Thomas M. Maddox, MD, SM, professor of medicine and vice president of digital products and innovation at BJC HealthCare, Washington University School of Medicine, gave a presentation on new technology advances at ACC.24, the annual meeting of the American College of Cardiology (ACC). He then spoke with Cardiovascular Business, providing insights into cardiology's future. "Digital health is really starting to mature and become a key part of how we're going to practice cardiology going forward. There are some newer ways that we are collecting data about patient's health, and particularly when patients are not with us in a clinic or a hospital. So we are monitoring them at home and making sure that we're not missing early signs of disease worsening or disease progression," Maddox explained. " American Heart Association MedTech Innovator LSI Heart Rhythm Society #epeeps

8

2 Comments

Big healthcare AI news yesterday as Tempus AI received the first-ever CPT code for an algorithm-only analysis of previously-sequenced, laboratory developed test transcriptomic data. TLDR: AI molecular phenotyping has a new business model. The PurISTSM test uses RNA profiling to identify molecular subtypes in patients with advanced pancreatic ductal adenocarcinoma, which may inform first-line therapy selection and management. While I'm skeptical that payer balance sheets are going to solve the "who pays for healthcare AI" problem, the reimbursement pipes for clinical AI are maturing. I am surprised to see more progress for usage-based vs. performance-based models given the many challenges with FFS economics. It will be interesting to see how payers assess the "medical necessity" and therefore reimbursability of clinical AI as these models come to market. https://t.co/eLe18SmihV

420

9 Comments

Unmet social needs could increase risk of costly surgical complications by 2-3x in some circumstances. Assessing and addressing SDOH factors is thus becoming mandatory under a variety of VBC payment models and episodic bundled payments. RevelAi Health's use of LLM-powered voice AI for specialty care management, social needs screening and closed loop social referrals with integration with Findhelp is delightfully seamless and groundbreaking at the same time - with far reaching applications. Very proud to share this simple demo and contribute to this work pioneered by Christian Pean M.D., M.S. and Hadi Javeed.

17

Artificial Intelligence is revolutionizing healthcare, bringing amazing advancements in diagnostics, treatment, and patient care. As these technologies evolve, it’s crucial to ensure they’re used safely and ethically. Here’s a look at where we are and what lies ahead: 🔍 Current Regulatory Landscape: In the U.S., the FDA and HHS are key players in overseeing AI-driven medical devices, ensuring they’re safe and reliable. Across the Atlantic, the EU’s AI Act, which kicks in August 2024, sets rigorous standards for high-risk AI systems, including mandates for transparency and ongoing risk evaluations. 🌍 Global Efforts and Recent Developments: The EU’s AI Act is a trailblazer, aiming to balance innovation with patient protection. The recent U.S. Executive Order on AI emphasizes the commitment to safe and trustworthy AI, with the White House AI Council spearheading cross-sector initiatives. ⚖️ Challenges and the Path Forward: With AI advancing rapidly, regulators face the challenge of keeping pace while ensuring safety. There’s a clear need for international cooperation to create unified standards. Moving forward, expect greater focus on transparency, accountability, and ethical practices. As AI becomes an integral part of healthcare, aligning regulations with technological progress will be crucial. Collaboration among government bodies, industry leaders, and global organizations will help ensure that AI innovations enhance patient care while upholding the highest standards of safety and ethics. Let’s stay engaged and informed as we navigate this dynamic landscape. Together, we can foster AI advancements that improve patient outcomes and uphold strong ethical standards. 🌐💡 #AI #Healthcare #Regulation #Innovation #PatientCare #Ethics #GlobalStandards #HealthTech #AIRegulation #FutureOfHealthcare #MedicalAI #AIInHealthcare #TechPolicy #DigitalHealth #AICompliance #HealthcareInnovation Read More: https://lnkd.in/g42FhhY3

1

Dr. Pearl, you’ve summed it up perfectly: “The combination of empowered patients, dedicated clinicians, and generative AI can transform American medicine and improve the health of our nation.” — Robert Pearl, M.D. This vision is not just inspiring—it’s essential for the future of healthcare. If you’re passionate about the intersection of AI and medicine, I highly recommend reading Dr. Pearl’s latest book, ChatGPT MD: How AI-Empowered Patients & Doctors Can Take Back Control of American Medicine. Let’s work together to create a brighter, healthier future for all. #HealthcareInnovation #GenerativeAI #Medicine Tina Gill

42

10 Comments

In a recent STAT interview, FDA Commissioner Robert Califf highlighted a crucial challenge in AI for healthcare systems: "Developers can’t fully guarantee the algorithm’s performance... an AI model’s performance evolves over time and performs differently in different hospitals’ patient populations, unlike a drug. And that model drift is 'happening in health systems.' So, for a health system to say, ‘This is just a manufacturer [problem],’ then the whole thing will break down and won’t work." https://lnkd.in/deHW2_PX While the debate over who bears responsibility continues, it's becoming increasingly clear that the liability for ensuring patient safety will ultimately rest with health systems. This aligns with one of the critical domains of the CLINICAL AI SOCIOTECHNICAL FRAMEWORK, developed during my PhD research: "Monitoring and Support." This domain addresses explicitly the complexities of building, purchasing, and implementing AI systems in clinical environments and includes: User Experience and Feedback: Continuously evaluating and incorporating user feedback to identify areas for improvement. Data Drift and Shift Management: Monitoring and managing changes in data patterns over time to maintain AI model accuracy and relevance. Updates and Review: Regular updates and periodic reviews are implemented to meet emerging needs and technological advancements. Continuous Education and Support: Providing ongoing training and support ensures users can effectively utilize AI systems. Audit and Surveillance: Conduct regular audits to assess AI systems' performance, compliance, and security post-implementation. To assist with these assessments and planning, we developed a checklist available for free here: https://lnkd.in/dHmbAt8s Also, here is a pre-print on the checklist: https://lnkd.in/gKnGxcFH

75

9 Comments

Exciting news from Sheba Medical Center, Tel Hashomer where they developed an AI algorithm to predict pulmonary embolism risk at hospital admission! According to a report in the Journal of Medical Internet Research, this groundbreaking tool was trained on medical records from over 46,000 patients. Led by Prof. Gad Segal and a team from Sheba Medical Center and Ben Gurion University the study demonstrates that the random forest algorithm can accurately predict pulmonary embolism risk based on patient history available upon arrival in the emergency room. Some takeaways: ✔ The AI model has been trained on a large dataset, enhancing its reliability. ✔ Sheba Medical Center plans to incorporate this model into routine clinical use, not just R&D pilot. ✔ The tool represents a significant advancement in AI-driven healthcare solutions, a reflection of a growing trend of AI integration into clinical practice. ✔ This innovation could lead to better management and outcomes for pulmonary embolism patients, a life saving impact. I am proud to be associated with Sheba, as an advisor to Sheba Longevity Center, the world’s first dedicated longevity clinic in a public hospital (read more in the comments). It is a great project and longevity resource for the community, spearheaded by Professors Tzipi Strauss and Evelyne Bischof, MD, PhD. While aging research and longevity startups are super important and cool, it is the application of modern technologies in actual clinical settings which makes an immediate impact on society. Sheba Medical Center is truly leading the way in adopting cutting edge technologies, like AI, and pushing forward pioneering ideas, like longevity! #healthcare #longevity #artificialintelligence Image credit: Sheba Longevity Center

66

3 Comments

Enjoyed giving this presentation today for Open Data Science Conference (ODSC) on Developing Equitable #AI Diagnostics. #ODSC2024 My slides can be found here: https://lnkd.in/gAbT4Yve

7

1 Comment

Breaking complexity to more manageable steps can allow #AI to be deployed in a controlled, responsible way: “#Systemengineering decomposes complex tasks into modular steps, enabling tasks like #clinicaltrial matching, clinical note generation, and automated coding through interconnected #LLM modules and human-in-the-loop validation.”

6

1 Comment

Get to know Dr Aengus Tran, co-founder and CEO of Harrison.ai, a Microsoft for Startups company, specializing in #AI for #medicalimaging. In this interview from Axios, hear what he has to say about challenges in healthcare; get to know more about his athletic pursuits; and read about his favorite tech device. Bethany Cordes Sinead Hewson Jacques Gilbert Dimitry Tran Randee Stapleton #founder #interview #healthcare #diagnostics #AI #AIinHealthcare https://lnkd.in/eNuYv7Vn

53

1 Comment

Healthcare is making exciting new strides in the realm of AI. Mike White, Healthcare Principal at Alexander Group, explores how AI is reshaping MedTech GTM strategies, navigating challenges while unlocking opportunities. Discover key insights into the future of AI in the healthcare industry! Read Mike’s full Spiceworks article here: https://lnkd.in/g4z4yZDs

17

Transformer architecture revolutionized the field of NLP and brought about the likes of ChatGPT. Now you have an interactive tool to understand how it works! The video below by Eric Vyacheslav provides a brief walkthrough of an interactive tool developed by a team from Georgia Tech and IBM Research. It demonstrates how transformers process input to generate output (i.e. how ChatGPT takes your prompt and produces a seemingly logical response). The tool also includes explanations for embeddings, transformer blocks, and output probabilities, allowing anyone to explore these concepts no matter your current understanding! 📊 Interactive Visualization: https://lnkd.in/e9Tz26ec ✍️ Paper by the Team: https://lnkd.in/ezeA4tf4 🎥 Great Walkthrough Video by Elvis Saravia: https://lnkd.in/evwHDJ_n Stay nerdy my friends!

50

4 Comments

Bank holiday fun: This is a nice post from Graham Walker, MD describing a heuristic that can be used to expand the range of diagnoses a clinician might consider when reviewing a patient, and how it could be used as an LLM use case. Given the incredible flexibility of LLMs and the ease with which this can be tested, I've created a GPT (ChatGPT plus sub required) that allows you to try it out. https://lnkd.in/eBAWsfn5 If you do not have access to GPTs, you can find the prompt itself on github: https://lnkd.in/erdVqcFs DISCLAIMER: Do NOT use this in a clinical setting! It's just a fun demonstration of how role-based instruction prompting can deliver helpful results. I'd love to hear your feedback. #ArtificialIntelligence #DigitalHealth #GenerativeAI #GENAI #LLM #Prompting #PromptEngineering #ClinicalPromptEngineering #GPT #GPTs #ChatGPT

8

3 Comments

Healthcare Innovation News for the week of Oct 18, 2024: · Healthcare Venture Studio Industry Report released by HDIG and Lawrence, Evans & Co., LLC - information on trends and leaders. HCVS's have become the most efficient innovation model to quickly grow ideas and IP into unicorns. Hospitals and Corporates are partnering, creating or investing into Studios. This leading industry report identifies, clarifies, and provides insights and contacts to every HCVS in the US. Review a copy here https://lnkd.in/g9nGiky5 and subscribe to HDIG for access to all contacts and detailed investments and performance. · GEMMA Biotherapeutics secures up to $100 million in funding · Health2047 (HCVS) Portfolio Company Zing Health Raised $140 Million · Sunbird Bio Secures $14 Million in funding · Purespring Therapeutics Raises $105 Million in Series B Financing for Gene Therapy Treatment of Kidney Diseases · Oshi Health Raises $60 Million for Virtual GI Platform · Parakeet Health Raises $3 Million for Expansion and Development · Pip Care Raises $5 Million in Funding For Surgical Care - The round was led by A1 Health Ventures, with participation from existing investor UPMC Enterprises. · RadiantGraph Receives $11M Series A Round Read about these and all the news https://lnkd.in/g4kp6unx Meet with HDIG team at the follow events: - Oct 16 - McDermott Will & Emery Hospitals and Health Systems Innovation Summit 2024 #HHS24 - Oct 20-23 - HLTH USA Las Vegas NV - Dec 3 - Bamberg Health Innovation Atlanta GA - Jan 13-15 - J.P. Morgan San Francisco HDIG Search/HDIG Partners/HDIG Funds HDIG is an ecosystem to make healthcare innovation more efficient. HDIG Search: a searchable database of hospital healthcare venture, innovation centers, tech transfer offices. It assists you in understanding the landscape of partnerships, performance, and contacts. Early-stage companies use it to find partners and funding for their pilots, or even professional firms or tech companies depending on their needs. HDIG Partners: Consulting & valuable resources for hands on insight, valuation & improvement. HDIG Funds: Capital investment into pilots & start-ups Launched with nearly 10,000 companies, hospitals, & funds, get full access for a special subscription & you can cancel at anytime, this would include building your profile. Learn more about our Acceleration Pilot Program, paid pilots, reverse studio model & other research & services. #healthcareinnovation #hospitalinnovation #innovationcenters #HDIG #venturestudios #startups

7

I have a strong affinity for the intersection of SaMD (SaaS As A Medical Device) and ML/AI. I see it as a vital part of the Healthcare ecosystem for some of the following reasons. 1. Connectivity between hospital systems, clinics and pharmacies 2. Enhanced diagnostic and clinical reporting capabilities 3. Payer and payee billing and insurance 4. Patient access to medical history 5. Innovation in drug discovery 6. Remote monitoring of patients I am impressed with tools like Cerner, and Epic. At this point it is hard for me to imagine life at any hospital with out them! I am excited about companies like Segmed, that democratize and empower researchers by providing a self-serve data discovery platform. Diagnostic tool platforms like Capsule Endoscopy in digestive healthcare are groundbreaking in offering physicians valuable clinical information informing patient health outcomes. There are concerns that I have over the adoption of SaMD and ML/AI in the Healthcare ecosystem. These issues are important for both the medical and venture communities to address and mitigate. 1. Bias introduced through ML models that are not inclusive 2. Interoperability lags 3. Cybersecurity risks 4. Diagnostic accuracy, and ultimate reporting responsibility When looking at startups in the exciting category of SaMD/ML/AI, I have much to consider. I am enjoying this journey in helping move it forward! Please see this informative article on SaMD/ML/AI in the digestive space for an enlightening read: https://lnkd.in/emXHQtKW Miguel Mascarenhas-Saraiva Cerner Healthcare Private Limited Epic Segmed, Inc. Please Follow Purelite Ventures: Purelite Ventures, NanaYaa Thomas, MBA, CLT

39

6 Comments

On #FederatedLearning for #AI models: “FL models often struggle with generalization across institutions due to non-uniform data distributions (non-IID), where variations in data size, labels, features, and quality across different sources complicate model generalization, and a lack of standardized data harmonization practices.… To overcome these barriers, the paper recommends improving model transparency, using adaptive frameworks, reducing communication burdens, and addressing statistical and system heterogeneity.”

3

1 Comment