Arvind Srinivasan

Meta's Llama 3 dropped just a few days ago, but already there are close to 1,000 variants publicly available on Hugging Face. Most people read base model evaluations and think they apply to all these deployments, but that ain't the case. Or at least the extent to which it's true is poorly understood. I want to see more eval work focus on the performance and risk shadow cast by base models on their downstream progeny. Basically, if base models aren't being used as-is, do their evals matter? Or are they poorly predictive of what will come of them post-fine-tuning? I don't have time/resources to work on this right now, but I'd love to see someone take this project on: across all variants posted on HF, how do evals on fine-tuned versions compare to evals done on the original base model? What kind of relationship is there between the two? Is the latter a constraint or directional anchor for the former? The best set-up here needs to find some way of ignoring some amount of noise (e.g., devs creating their own copy but not doing any fine tuning or doing some kind of terrible fine tuning) in order to focus on more serious attempts to fine tune for stronger performance on specific use cases, stronger safety and security, or something else. This question is increasingly important because fine-tuning is becoming easier and faster to do. That means it's more and more likely that developers will tune models to their own liking. Evals done on prior upstream versions are potentially meaningless... 'potentially' is the key word because we don't have a good understanding of the persistence of performance and risk, the elasticity of learning. By the way, if you're interested in working on hard questions around LLM evaluation, like these ones, we're hiring! Our Trust & Governance research team is looking for a Staff Applied Research Scientist. Come help us tackle these challenges! Link to the job posting in the comments. #artificialintelligence #trustworthyai #llm

51

9 Kommentare

Who will get crushed by Claude LLM coming to iPhones? Will existing apps (and companies) get steamrolled by this? For those that don’t know Claude is a suite of 3 LLM models: Haiku, Sonnet and Opus. - Opus knocked ChatGPT off the leaderboard on capability when it was introduced. - It can hold a massive 160,000 words “enough for a user to paste in a weighty novel and ask follow-up questions” - Claude is multimodal so you will be able to take pictures or upload images from your iPhone and include that in your prompt Use cases * After a meeting, a business user could snap a photo of a whiteboard diagram and ask Claude to summarise the key points, making it easier to share and act upon important information. * Similarly, a consumer could take a picture of a plant they encounter on a hike and ask Claude to identify the species and provide more information about its characteristics and habitat. ** think the Picture This app ** What other uses do you see for this? https://lnkd.in/gdExfpx9

6

Large Language Models (LLMs) as super lossy compression. I have been thinking a lot about demystifying what LLMs do because lots of y'all's are very mystified. In the linked article, I try to reframe them for you, hoping it will help you understand what they good for and what they are not good for. Side note: I am still looking for my next adventure. If communicating complicated concepts so that regular people can grasp them is something you need, you should message me. I write code and stuff too and am usually fun to be around. #WrittenByMe

6

3 Kommentare

Tembo is leading in the 'Postgres as a Platform' movement. Their most recent blog post(s) below do a pretty good job of explaining why 'Postgres as a Platform' is rapidly becoming an anecdote for database sprawl.

8

1 Kommentar

Meta researchers develop ‘thinking’ LLMs  Meta researchers just introduced a new method called Thought Preference Optimization (TPO) to train large language models to ‘think’ before responding to general instructions — not just reasoning tasks. The details: →TPO prompts models to generate internal thoughts before responding to user instructions, similar to how humans think before speaking. →The AI's thoughts are kept private, with only the final answer shown to users with the AI using trial-and-error without direct supervision to optimize outputs. →TPO outperforms standard models on key benchmarks for non-reasoning tasks like marketing and creative writing but declines in math-related tasks. →The approach builds on the recent OpenAI ‘Strawberry’ research and o1 model release, which takes time to reason. Why it matters: Meta’s own Yann LeCun may scoff at the notion, but this method shows the potential of AI to ‘think’ for broader tasks than just math and reasoning. By allowing models to learn useful thought processes independently, TPO could also enable more capable and flexible AI assistants across a wider range of domains.

6

1 Kommentar

With Redpanda Data's recent acquisition of Benthos I've gotten some questions about how it compares to Arroyo. In short: Benthos is *stateless*, while Arroyo (like Flink or RisingWave) is *stateful*. This may sound small, but adds up to a fundamental difference in how these systems are designed, built, and used. In a streaming engine, statefulness combines two inter-related features: * Repartitioning data (GROUP BY in SQL) * Remembering previously seen events With just these two features, we can support most of normal SQL (window functions, joins, aggregation, etc.) plus time-oriented streaming operators (time windows, event-time semantics, watermarks). Stateless engines on the other hand are limited to simple transformations and filters (SELECT and WHERE in SQL terms) that operate on individual events. But all of this is a tradeoff; stateful systems are much harder to operate. So if you don't need joins, aggregates, or windows, a stateless system can be a better choice. They also work well as infra glue, routing data between different systems—Benthos in particular has a huge connector ecosystem. For a deeper dive on this, I wrote down some thoughts on our blog:

57

4 Kommentare

At the end of my PhD, our lab at UC Berkeley (with Joseph Gonzalez Ion Stoica) was approached about developing courses with Andrew Ng about our research on memory management for agents. We squeezed in developing a course in between starting a company, fundraising, hiring, and building product (https://lnkd.in/dR9fBxEf), and now we’ve finally released the first course on building LLM agents with memory! Memory is essentially how we program the context window of LLMs to get agents to know what we want and do what we want. This is critical for moving from simple chatbots or single-action workflows (the majority of LLM applications today), to agents which can execute complex sequences of actions, learn over time, and act in a deeply personalized way. You can use this course to understand how agents use memory to reason and learn over time, and how to customize this behavior for your own applications — while also learning about some of the academic work and ideas behind MemGPT (with Charles Packer Kevin Lin) and Letta. This course also is the first time we’ve released tutorials on how to implement advanced customization in Letta, such as customizing the algorithm for memory management (as well as how memory is represented) beyond what was in the original MemGPT paper, integrating with both custom tools and tools from other providers (with support for LangChain, CrewAI, and Composio), agentic RAG, and multi-agent. Thanks so much to Geoff Ladwig, the DLAI team, and the Letta team for helping develop this course 😊 You can checkout the course below, as well as the tutorial notebooks on our github! https://lnkd.in/gsZyTuEV

131

8 Kommentare

This new little product demo is awesome! Our new Rev AI Transcript Assistant will save you TONS of time. Easily pull out nuggets of info from your transcripts, making valuable content more accessible than ever. https://lnkd.in/gzzFpcDa #transcription #asr #accessibility #transcript #summarization #ai #assistant

31

2 Kommentare

To ensure the success of Generative AI, effective expectation management is crucial. It's important to approach these technologies with caution, avoiding the assumption that language models alone will deliver extraordinary results. While our capabilities have significantly advanced compared to a few years ago, it’s prudent to take measured steps forward, one at a time.

27

🔥Hot topic - is AI good or bad for climate change? Great convo at Sifted Summit today with Andreas Brenner at Jua.ai, Raphael Güller at SWEEP , Zoe Balmforth at Pivotal Future and Dominic Vergine at Monumo The answer is…both. Lots of concern around the near term consumption spikes. But lots of optimism around how AI can unlock step change solutions across material design, weather forecasting and risk, biodiversity conservation and more. Ultimately AI increases the urgency to increase renewable and clean energy capacity. Demand isn’t going away…we need more energy to unblock the generational breakthroughs.

9

Last year, Huggingface deprecated their InferenceApi client and switched inference calls to InferenceClient. Simple enough. But see, Langchain's HuggingfaceHub package still called InferenceApi. So anyone running langchain.llms.huggingface_hub received an future warning error that InferenceApi was depracated and to use InferenceClient instead. The problem was that unless you went searching around into the details of the code langchain.llms.huggingface_hub package, you had NO idea what this error meant and whether it was an important warning to heed. So one would think Langchain would resolve this quickly, right? No. This was not resolved universally until a January 2024 update. Until then, people were using patches to get around this error. AND if you're thinking, "Wow! Well, glad that's no longer a concern" think again because out of curiosity, I asked ChatGPT 4o list imports for me to create multi-agent AI using langchain and inferencing the mistral API and wouldn't you know it, it suggested I use the deprecated InferenceApi client. Now, can you use the InferenceApi client and it still work? Sure. However, that future warning error gives lots of engineers who have not seen that error before pause AND it would likely be an issue when considering if one's code is production ready. When one AI company changes a client name and another AI company is using their own client to call on the original AI's client, this creates another dependency. One that a LOT of engineers building AI are not aware of unless they get used to seeing these errors over and over and over again. Little things like this are keeping great AI model demos from becoming great AI products.

45

6 Kommentare

In the future, all efforts will on data curation and engineering. All model training and Fine-tuning will be standardized. That doesn’t leave much to do for AI engineers :)

2

If ChatGPT was AI's "IPhone Moment", by now we're well down into the IPhone 5s moments. Lots of minor tweaks, maybe a new button or missing connector, still lots of hype for some reason, but fundamentally it feels pretty much the same. That said, don't believe the naysayers that see this as the beginning of a decline. I'm still very bullish on AI, it's just that the interesting parts are shifting from who can throw the most compute, and raw data (and legions of fundamental researchers) at the problem to who can make the advances needed to expand the technology to new uses. (Note added at writing, I just realized I'm effectively saying the opposite as what Sam Altman said this week which was "assume it will get way better" instead of find ways to live with current performance. Good. I disagree and the previous deep learning cycle backs me up). All that said, I wanted to share some interesting AI stuff that's on my radar this week that's not about new model releases (mostly): https://www.vals.ai/ Vals.AI is a startup making application specific benchmarks. Good benchmarks and better performance go hand in hand, getting out of theoretical evaluations and in to something practical is going to be the future. (I think the same thing about data: https://lnkd.in/eNepJ7Ys) https://lnkd.in/eHxbBE2e Eq-Bench is another benchmark that I hope is going to help show how silly many modern evaluations are. It achieves great predictive power for how models perform on other tests and in human evaluations by asking them to rate the emotions in dialogue samples. It's a great little test but also has nothing to do with most real applications, and sort of implies that neither do most of the other tests. https://lnkd.in/eFkAXYb7 IPEX LLM by Intel. This has been around for a while but is getting more interesting as Intel releases new GPUs and there is a push to do model inference faster and more efficiently. Check it out as an alternative to Hugging Face for LLM inference. https://lnkd.in/eFJdV_AB Pile-T5 by Eleuther AI. This work takes an old model and an old data set and looks at what happens when you train it for twice as many steps. Most interestingly, they release (if I counted right) 200 checkpoints along the way. This is mostly unheard of for big models and I'm hoping (as are they) it's going be lead to some interesting results related to interpretability, (which despite what you may have heard isn't really possible with LLMs yet). For example https://lnkd.in/eWQZCbu2 So, boring time this week for new mainstream models, lots of other cool stuff going on. Don't give up on AI yet.

14

1 Kommentar

[Hot News] 🖥️ Game-Changing: Claude AI Now Controls Computers Like a Human Anthropic has just revolutionized AI interaction by enabling Claude 3.5 Sonnet to directly control computers, marking a significant leap in AI automation capabilities. This breakthrough allows the AI to navigate screens, move cursors, and execute complex tasks autonomously. The new "Computer Use" feature empowers Claude to perform a wide range of operations, from taking screenshots to filling out forms and analyzing data. While currently in beta and primarily targeted at developers, this advancement signals a fundamental shift in how AI can interact with computer systems to enhance productivity. This transformative capability opens up endless possibilities for automation and efficiency, from streamlining complex workflows to handling repetitive tasks across multiple applications. The system's ability to understand context and adapt to different software environments makes it a powerful tool for modern digital workspaces. Key Takeaway: This breakthrough transcends traditional Robotic Process Automation by introducing truly intelligent computer interaction. We're entering an era where providing instructions replaces implementation - a genuine AI assistant that can understand and execute complex computer tasks through simple commands, fundamentally transforming how we interact with technology. Follow me and CMRS Group (HK) for more exciting AI news! Read more : https://lnkd.in/gbjXNEwv #AITransformation #ClaudeAI #TechInnovation #AIAutomation #FutureOfWork

7

This article goes into how you can train a 70-billion-parameter large language model #LLM at home. It’s a great read (not just a PR fluff) for all technical people, and it explains how quantization, Low-Rank Adaptation #LoRA, #QLoRA, Distributed Data Parallel #DDP, Fully Sharded Data Parallel #FSDP, and Half-Quadratic Quantization #HQQ work. I’ve learned a lot, and this is why we should support open-source projects: https://lnkd.in/getNAzDb

7

1 Kommentar

Fascinating to see how traditional concepts in computer science like virtual memory are now being deployed to solve new problems in agentic architectures like persistence beyond finite context length. To the human brain that inspired much of computer science and to computer science that now keeps paying it forward.... 🚀

14

Awesome example of how reproducible dev environments with Jetify Devbox speeds up developer onboarding: "Linus demonstrated how he used Devbox from Jetify to significantly reduce the onboarding time for his students – from days to just a few hours – by setting up his computing environment in a re-runnable way."

4

Senior engineers lose money and responsibility at Meta partly because of an unwritten rule. Candidates are expected to not follow their interviewer's instructions to SUCCEED. Yes, you read that right. That blew my mind. There are other common reasons for downleveling as well. Some of these are expected but others are surprising. Kevin Landucci + Kashish Singhal, loved the simple practical takeaway for candidates. And the full episode rocked! Subscribe to ApplyPass to get more tips in your feed 🛎

20

3 Kommentare

Smaller and more informed (#RAG/LORA) models are faster and more useful in business practice. You shouldn't treat #LLM's as a general purpose silver bullet or you will have never create a product of any value. Always pick a composite of the right model and data for the problem. This three point strategy (Model, #Data, Problem) will help you create better results for you and your consumers. Don't be afraid to chain results together to generate a larger encompassing solution.

3