Deciding which LLM to use is hard. Trying to factor in the legal, ethical, or risk considerations of each model is even harder. A lot of this information is hidden deep in 50+ page technical reports, and often the key information necessary to determine whether a model is appropriate to use a task isn't disclosed. We're looking to help solve that problem by releasing our first ever Model Transparency Ratings. We analyzed the public documentation of the top 21 LLMs against the requirements of General Purpose AI Models from the EU AI Act and created a scoring criteria to identify which models may be riskier to use for AI products in the EU. ------- A few insights from our first set of ratings: 1️⃣ Models have been getting LESS transparent over time. Many providers like Meta, Cohere, and Mistral are less transparent about their newer models than their older ones. 2️⃣ Very few LLM providers disclose their data sources. I'd argue that understanding the data sources is MORE important to understanding the risks of an AI system than the model architecture, but we see decent model transparency, and poor data transparency. 3️⃣ "Open Source LLMs" do generally better, but we had to separate the idea of 'open weights' and 'open data' to better distinguish the risks and expectations. ------- Check out our ratings at: aimodelratings.com To read more about Trustible's methodology, check out our blog post here: https://lnkd.in/gShTuTGx We'll be expanding our ratings to include both new models, and include additional risk criteria. Share what you'd want to see in future versions of our ratings in the comments!
Denis Peskoff
8mo
Joseph Barrow curious about your thoughts
Education Innovation Leader | K-100 Learning Strategist | EdTech Investor & Entrepreneur
8moYusuf Ahmad