Simon Hay

From Cecile G. Tamura on Facebook: MMSci: A Multimodal Multi-Discipline Dataset for PhD-Level Scientific Comprehension - The paper presents a new multimodal, multi-discipline dataset (MMSci) containing 131,393 high-quality peer-reviewed articles across 72 science disciplines from Nature Communications journals. - It includes 742,273 figures along with captions averaging 153 words, providing rich multimodal scientific data. - A benchmark is constructed to evaluate models' understanding of PhD-level scientific knowledge across disciplines, with tasks like scientific figure captioning and visual QA. - Evaluations reveal deficiencies in current models' capabilities to accurately interpret scientific figures and content, even for large models like GPT-4V and GPT-4o. - The dataset is explored as a training resource by constructing over 1 million visual instruction-following samples and through pre-training on interleaved article text and figures. - Experimental results demonstrate the effectiveness of the constructed training data in improving model performance on the benchmark and on a material generation task. - The work helps advance multimodal scientific comprehension and the development of AI-based research assistants. The models, data, and code are open-sourced. abs: The rapid advancement of Large Language Models (LLMs) and Large Multimodal Models (LMMs) has heightened the demand for AI-based scientific assistants capable of understanding scientific articles and figures. Despite progress, there remains a significant gap in evaluating models' comprehension of professional, graduate-level, and even PhD-level scientific content. Current datasets and benchmarks primarily focus on relatively simple scientific tasks and figures, lacking comprehensive assessments across diverse advanced scientific disciplines. To bridge this gap, we collected a multimodal, multidisciplinary dataset from open-access scientific articles published in Nature Communications journals. This dataset spans 72 scientific disciplines, ensuring both diversity and quality. We created benchmarks with various tasks and settings to comprehensively evaluate LMMs' capabilities in understanding scientific figures and content. Our evaluation revealed that these tasks are highly challenging: many open-source models struggled significantly, and even GPT-4V and GPT-4o faced difficulties. We also explored using our dataset as training resources by constructing visual instruction-following data, enabling the 7B LLaVA model to achieve performance comparable to GPT-4V/o on our benchmark. Additionally, we investigated the use of our interleaved article texts and figure images for pre-training LMMs, resulting in improvements on the material generation task. The source dataset, including articles, figures, constructed benchmarks, and visual instruction-following data, is open-sourced. #AI #computervision #patternrecognition #LLM #language #computation

End of an era? Just published the final brief in the series on guidance for working with 2020 Census data products. For those who use the data, I hope the series is helpful. #2020Census #Demography #Data #DataScience #DisclosureAvoidance #DifferentialPrivacy (Link in comments)

36

1 Comment

This study in AJCP investigates the adoption and perception of large language models (LLMs), such as ChatGPT, among pathologists. https://bit.ly/3ywLwBX

3

An innovative methodology of ours (converting NDVI data into vegetation phenology indicators, e.g., the onset of greening (OG) in spring and the onset of dormancy (OD) in the autumn), which is introduced and used in our paper (https://lnkd.in/edsNFcBN), was not highlighted too much as the focus of the paper was not on the methodology. So, I decided to put it in my GitHub repository and to provide instructions and example usage to promote it. Methods introduced in the literature that also use the NDVI data to derive OG and OD assume that bud break for a given pixel occurs at a specific NDVI threshold that is greater than the observed NDVI minimum value in winter and early spring. For example, OG is commonly defined as the day of the year when NDVI reaches the NDVI minimum plus 20% of the NDVI range or the critical NDVI value determined from the long-term average annual curve of NDVI. The same also applies to OD. Here, however, we introduce an innovative method that determines this critical NDVI value by cumulatively plotting NDVI. This allows us to show that the curve consisted of three parts that could be approximated linearly and that correspond to different growing periods. The critical NDVI values were then determined by calculating the intersection of the bisectors of the first two (in the case of OG) or last two (in the case of OD) lines and the cumulative curve rather than an arbitrary criterion of minimum NDVI plus 20% of the NDVI range or long-term average critical NDVI. Any remaining uncertainties in our OD and OG estimates are therefore systematic, which can be important in many research projects where a trend analysis is planned. On the other hand, our introduced method, namely the logistic function derivative of the NDVI (LFD-NDVI) curve method, is an innovative algorithm that could operate independently for each year and pixel. This is important because classical methods, e.g., Piao et al. (2006) https://lnkd.in/e7zygQyV, typically calculate a critical long-term NDVI value for the entire period and then determine the timing for approaching such a critical value in all individual years. The LFD-NDVI method uses the first and second derivatives of the fitted logistic function of the cumulative NDVI curve to determine OG and OD. https://lnkd.in/eeKKemxt

48

1 Comment

Our webinar on resource management in the South African district health system happens to coincide with our SA president signing the NHI Bill.... and clearly resource management is a key area of health systems strengthening for any UHC system, including NHI! I look forward to the debate!

13

Stay up to date with the latest research in Measurement, Evaluation, Statistics, & Assessment with the latest ESM MAD Blog! #ELPS #ESM Check out our latest blog by Anthony Schmidt here: https://lnkd.in/eGVHSEUV

6

My background isn't statistics or data science - it's GIS and data analytics 🗺 For a long time, I shied away from using statistics in my analysis because I felt: - I didn't have the right expertise - I didn't have the right tools - I didn't know how to communicate statistical outputs to a non-technical audience Tomorrow, I'm holding a webinar about how you can overcome all of this! I'll be sharing how you can build spatial statistics into your analysis for more quantifiable insights. We'll be investigating problems including: 📶 How can you develop an OOH strategy based on the spatial clusters of billboards? 🚄 Do house prices in cities always increase in areas with better public transport? 🏡 How can you assess wildfire risk from a wide range of data sources? We'll be doing all of this code-free and with a MASSIVE amount of data! Sign up below! 👇

101

3 Comments

ICYMI Federation of American Societies for Experimental Biology (FASEB) DataWorks! offers a range of resources to support data management and sharing in the biological sciences. The DataWorks! Help Desk has several resources to help you find the most appropriate repository for your data - check them out BEFORE the next grant deadline! ⬇️⬇️⬇️ #FASEBDataWorks #Datamanagement #datasharing #datarepository #FAIRdata #skillbuilding

7

Stay up to date with the latest research in Measurement, Evaluation, Statistics, & Assessment with the latest ESM MAD Blog! #ELPS Check out our latest blog by Dr. Austin Boyd here: https://lnkd.in/eved9CYB

6

Introducing Butterfly Plots in #SigmaPlot16 🦋​ Effortlessly compare datasets with Butterfly Plots, ideal for healthcare, climate studies, and surveys. Spot trends & analyze data side-by-side with SigmaPlot 16’s latest feature! ​ https://lnkd.in/e3pQnF63 #DataViz #Analytics #DataScience

3

1 Comment

🌟A Notable Paper Published #AvianInfluenza: Lessons from Past Outbreaks and an Inventory of #Data Sources, Mathematical and #AI Models, and Early Warning Systems for Forecasting and Hotspot Detection to Tackle Ongoing Outbreaks Free Access: https://lnkd.in/gGx7CaUa

3

🚀 For anyone developing production-level generative AI applications, monitoring and evaluation can be one of the most challenging aspects. It often requires multiple tools for logging, versioning, cost tracking, evaluation, and more—making the process cumbersome. I recently discovered the Weave toolkit through the free RAG++ course offered by Weights & Biases, and I am truly impressed! 🤩 It’s incredibly easy to start with just three lines of code, and the interface is super intuitive. 💡 🔍 Here are some of my favorite features: Multimodal Support 📊: Weave handles various data types—text, images, and audio—and visualizes them all in one place. Seamless Version Control 🔄: It automatically tracks changes to prompts, temperatures, and code, making versioning effortless. Testing Tools 🧪: A wide range of built-in evaluation options for consistent and effective model testing. ✅ 💡 Additionally, Weave performs exceptionally well with Retrieval-Augmented Generation (RAG) applications, tracking crucial metrics like retrieval accuracy and chunk quality—all in one central location. 📈 I’ve been experimenting with it for a few days now, and I find it to be an incredibly useful tool! ✨ Check it out here: https://wandb.me/a 🌟

5

1 Comment

🗽 🚀 #SDSC24 NYC earlybird tickets have just landed!! Join the spatial event of the year for 2 days packed with expert-led talks, hands-on workshops and much more! Learn more and get your ticket 👇 #datascience #dataanalytics #dataengineering

5

The DHS Climate Change & Health Research Hub released the second post in its series on temperature data! It demonstrates how to link longitudinal heat exposure metrics to individual records in the DHS. #GlobalHealth #spatial https://lnkd.in/gRVp8naH

1

In the era of Reviews and the emergence of new methodologies, this here is a great resource for all. An insightful quick read from JBI. #EBPadvocacy #EBPinPhysiotherapy #EIDM #AfricalovesEvidence

10

This is a great set of tools for professionals in public health and other fields to assess and begin to address the commercial determinants of health, health inequities, and other social and environmental problems (#cdoh, #commercialdeterminants).

1

check out our new PREPRINT where we review multiple imputation approaches for handling incomplete longitudinal data (including the case with higher-level clusters), summarize available guidance and illustrate implementation (in R and Stata) https://lnkd.in/gD_JznPd

14