Meenal Nalwaya’s Post

A few weeks ago Meenal Nalwaya who heads up multimodal LLMs at Meta and I were chatting about what is going on in the world of agentic infra globally. There have been a bunch of opinions on what agents are, what they can do, but there was no clear canonical architecture that was emerging. Even doing something as basic as a personal agent to book "a flight ticket to SFO tomorrow" needs the agent to know what "today" is, where "here" is to book something from here to SFO, my flight + seat + meal preferences, my FF number, my ID & credit card info, then once booked, schedule an uber ride, get my boarding passes & place it on my calendar, and ensure a pickup & a hotel once I land, and so much more. A simple task like this has so, so many complexities. We felt we needed to dig into what is the architectural framework needed to solve some of this. We also felt that this problem needs a 2-sided approach - you can build a bicycle but then you also need the roads to build bicycle lanes or it doesn't work. While we believe this utopian world where agents can book us a flight, a car, a hotel, is still not here yet, we wanted to write a series of articles to help us think about the opportunities and challenges ahead. This is the first part of a series of articles, feedback & comments welcome - link in the first comment - and hope you enjoy the Homer Simpson references :) part 1: "Endless Homers, Infinite Agents":

First and foremost, an agent must be aware of its own limitations—what it cannot do. When presented with a task, the agent should assess what it is capable of achieving using the available tools and context. It should also determine when to seek external agentic assistance, whether through recursive or iterative help.

If you’re interested in how you’ll work with AI in the future - AND are a Simpsons fan - this article is for you. Just like with humans, how agents share what they learn and how we interface with them, will govern the impact they have on their work. This is a great summary of the important points.

Hemant Mohapatra and Meenal Nalwaya published an insightful breakdown of what's missing in #agentic infrastructure and how we get from flashy demos to practical, functional systems. The article dives into the architecture needed for agents to move beyond copilots—covering key areas like orchestration, memory, and compliance. They highlight real challenges, like booking a simple flight, and explain why it’s not so simple: agents need to handle context, preferences, APIs, and even logistics like hotels and transportation. Orkes is mentioned as a tool simplifying multi-agent systems. It’s clear that solving these problems will require better infrastructure, not just smarter agents. If you’re building in this space, it’s worth a read. It unpacks where we are, what’s ahead, and what still needs solving. Check it out ⤵️

Meta Releases Llama 3.2 🚀 𝗟𝗹𝗮𝗺𝗮 𝟯.𝟮 the recent released version by Meta offers cutting-edge 𝗺𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗰𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀, supporting 𝗯𝗼𝘁𝗵 𝘁𝗲𝘅𝘁 𝗮𝗻𝗱 𝗶𝗺𝗮𝗴𝗲 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗳𝗼𝗿 𝗲𝗻𝗵𝗮𝗻𝗰𝗲𝗱 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀. From mobile-optimized lightweight text-only models (𝟭𝗕 & 𝟯𝗕 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀) to high-performance models vision LLMs (𝟭𝟭𝗕 & 𝟵𝟬𝗕 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀). Llama 3.2 ensures flexibility for 𝗼𝗻-𝗱𝗲𝘃𝗶𝗰𝗲 𝗔𝗜 like 𝘀𝘂𝗺𝗺𝗮𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻, 𝗶𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝗼𝗿 𝘁𝗲𝘅𝘁 𝗿𝗲𝘄𝗿𝗶𝘁𝗶𝗻𝗴 and 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 𝘁𝗮𝘀𝗸𝘀 like 𝘃𝗶𝘀𝘂𝗮𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 and tool integration. Models are available on 𝗹𝗹𝗮𝗺𝗮.𝗰𝗼𝗺 and 𝗛𝘂𝗴𝗴𝗶𝗻𝗴 𝗙𝗮𝗰𝗲 Read more about Llama 3.2 https://lnkd.in/gaCt7ZxC

𝗟𝗹𝗮𝗺𝗮 𝟯 !!!!! 😎 🔥 Talk about perfect timing! This morning, we were brainstorming cost-effective ways to fine-tune our models with lighter versions. Then, boom! Meta drops Llama 3, a next-gen 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 LLM that's turning heads. 𝗪𝗵𝘆 𝗶𝘀 𝗟𝗹𝗮𝗺𝗮 𝟯 𝗲𝘅𝗰𝗶𝘁𝗶𝗻𝗴? It's a beast! Performance rivals fancy, closed-door models, potentially topping our current 𝗚𝗣𝗧𝘀. 𝐂𝐨𝐬𝐭-𝐜𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬? Us too! Lighter models (like the 8B one) align perfectly with our budget-friendly goals. 𝐎𝐩𝐞𝐧-𝐬𝐨𝐮𝐫𝐜𝐞 𝐫𝐨𝐜𝐤𝐬! We can tinker and fine-tune for our specific needs, saving resources. 𝐁𝐫𝐚𝐢𝐧𝐩𝐨𝐰𝐞𝐫 𝐛𝐨𝐨𝐬𝐭! Improved reasoning, code generation, and following instructions could be a game-changer. 𝗟𝗲𝘁'𝘀 𝗽𝘂𝘁 𝗟𝗹𝗮𝗺𝗮 𝟯 𝘁𝗼 𝘁𝗵𝗲 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝘁𝗲𝘀𝘁! 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐭𝐢𝐦𝐞! 𝐋𝐞𝐭'𝐬 𝐬𝐞𝐞 𝐡𝐨𝐰 𝐋𝐥𝐚𝐦𝐚 3 𝐬𝐭𝐚𝐜𝐤𝐬 𝐮𝐩 𝐚𝐠𝐚𝐢𝐧𝐬𝐭 𝐨𝐮𝐫 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐆𝐏𝐓𝐬 𝐨𝐧 𝐨𝐮𝐫 𝐦𝐨𝐬𝐭 𝐢𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐭 𝐭𝐚𝐬𝐤𝐬. Cost crunchers unite! We need to analyze the financial implications of fine-tuning Llama 3's lighter models. Experimentation playground! Llama 3's open-source nature lets us play around with fine-tuning techniques and see if we can unlock even better results. 𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐥𝐞 𝐀𝐈, 𝐚𝐥𝐰𝐚𝐲𝐬! Safety first! We'll need to review Meta's safety tools (Llama Guard 2, Code Shield, CyberSec Eval 2) for responsible integration. We'll keep an eye on Meta's future plans for Llama 3 to see what exciting things are on the horizon. What do you think? Should we explore Llama 3 for our fine-tuning needs? Share your thoughts! #Llama3 #LLM #OpenAI #DCODEAI #FineTuning

Google Maps' revolutionary AI-powered upgrade 🗺 📍 Google Maps' recent AI-powered upgrade is a game-changer for navigation and user experience. The introduction of Immersive View for Routes allows users to preview their journey in a multidimensional view, whether they're driving, walking, or cycling. This feature, which is rolling out in major cities around the world, provides visual turn-by-turn directions and simulates traffic and weather conditions to help users plan their trips better. Moreover, the integration of Lens in Maps, formerly known as Search with Live View, uses AI and augmented reality to help users quickly find information about their surroundings, like nearby ATMs or coffee shops. This expansion to over 50 new cities enhances local navigation and discovery. The upgrade also includes Photorealistic 3D Tiles in Google Maps Platform, enabling developers to create immersive experiences. Additionally, the use of generative AI, powered by large-language models, generates text and suggestions from millions of saved places and reviews, enriching the search and recommendation process. These advancements underscore Google Maps' commitment to leveraging AI to provide a smoother, more detailed, and personalized navigation experience, setting a new standard for mapping services. Source: https://lnkd.in/eVYh85DY #GoogleMapsAI #ImmersiveNavigation #AIforDiscovery #3DTilesExperience #GenerativeAI

I’ve just had an exciting glimpse into the road ahead on Google Maps, Google Earth and Waze where we are bringing fantastic #AI-powered Gemini experiences to our flagship products. Drivers using both Maps and Waze will be able to benefit from a range of new features that will bring convenience to you entire journey... Before your drive ...  🚗 Explore for trips: plan your road trip with ease! Find restaurants, attractions, and more along your route. On the go ...  🌧 Weather Incident Reporting: see and report weather incidents like flooded roads, low-visibility roads, and more on your route to help other drivers. 🗣Waze Conversational Reporting: English-speaking Waze users can now report incidents using their voice — with no need to use a specific voice command or tap extra buttons — thanks to Gemini capabilities. When you arrive ...  🚙 Updated Arrivals Experience:  seamlessly navigate from your car to your final destination with parking info, walking directions, and Live View. To find out even more about Google Maps’ latest innovations, please see our global blog: https://lnkd.in/gNacRFPA #GoogleMaps #GoogleEarth #Waze #AI

This week, each day, I will publish my push for Technology At Work. Little nudges into a the future. Shorts, like the one today, will be from the interview with Ronnie Overgoor for 7DTV sponsored by loket.nl (see in the comments link to the whole interview). Tomorrow, it will be from the dialogue with Martijn Hamann, co-owner of Endeit Capital, growth Capital investor in Future of Work and AI and a super well connected human being in the tech space. #ai #fow Today the message is - and I will repeat it again and again- immerse yourself and your teams into the possibilities of AI and tech. A no-brainer...but in Europe a lot of tech isn't available (e.g. you CAN buy the Meta Glasses in the Best Buy in the USA but YOU CAN NOT buy them in a lot of countries in the European Union). We do not have self driving cars in our streets, although the technology is available. I understand why but the problem is that we, for that reason, are LIMITING our FANTASY and INSPIRATION about the future possibilities and applications. I loved the experience of a self driving taxi. I love pin ball machine in my mind that goes off experiencing the newest technology. It inspired, created new ideas, energized and challenged thinking patterns... At Eva Jinek's show EVA, Alexander Klöpping showed the easiness of cross passing the line of privacy with meta glasses. Oh scary 😱 ! But that is only one part of the story. It can also tremendously help in training environments to give suggestions, coaching, feedback and training footage for improvements. There are tons of ways to improve our skill levels with this single item, but most of us...are not thinking about that possibility because we don't see them...we don't use them..they are not on the desk, we might be too aware of the risks, and not exploring enough the opportunity. THAT NEEDS TO CHANGE. 🚀

Imagine if Meta were responsible for building Norway’s bridges and roads, collecting tolls, and profiting from our daily commutes. ⏬ That’s an odd thought.. why would we let them do that. We wouldn’t allow private companies to impose taxes on our physical infrastructure. So why do we permit them to control and profit from our digital infrastructure? Right now, our data is being 'taxed' by tech giants, fueling the development of increasingly sophisticated AI models. This raises important questions about who truly benefits from our digital infrastructure. It’s time to rethink how we manage and protect our data, ensuring it serves the public good rather than just corporate interests. To protect people’s data - we need to build “data bridges”. This is not possible with only a 1.09% budget in innovation.