Aravind Parameswaran

What if your queries required access to multiple sub-related data from different documents, for an LLM to parse and make sense of before returning a coherent and accurate output? We often ask complicated questions, right? 🔹Well to solve this problem, these researchers released Multi-Head RAG (MRAG) which leverages activations of Transformer’s multi-head attention layer, instead of the decoder layer, as keys for fetching multi-aspect documents. They are taking advantage of the fact that multi-head attention can learn to capture different data aspects. 🔹The interesting part is that MRAG is both cost-effective and energy-efficient. In other words, it does not require additional LLM queries, multiple model instances, increased storage, or multiple inference passes over the embedding model. 🔹Moreover, MRAG can be seamlessly integrated with existing RAG frameworks and benchmarking tools like RAGAS as well as different classes of data stores. 🔹The researchers provide a comprehensive evaluation methodology, including specific metrics, synthetic datasets, and real world use cases to demonstrate MRAG’s effectiveness. 🔸Do check it out! #genai #technology #artificialintelligence Click #linkedangle and follow for more content Set the 🔔 notification on my page, don’t miss a post!

181

10 Comments

10 TRILLION $ QUESTION: How much of a leap forward will ChatGPT5 be? If it’s only 20% or 30% or even anything less then 100% better, it will indicate that these models are topping out and we are shifting into a deployment phase with roughly the existing capabilities. This has massive implications for the foundational model builders and infra providers, as we are reaching diminishing marginal returns from new foundational models. The investment cycle in new models will slow down dramatically and it will be a race to the bottom on token prices. Most of the value will accrue to the application layer. We are still in the early days of deploying this technology and haven’t yet found the new ai first applications that create $100b+ companies, but we likely will soon. Maybe even $1tn or even $10tn. If we find out ChatGPT5 is just incremental, most of the energy and capital will likely shift toward building the application layer racing to win this prize. If ChatGPT5 is >100%, or even 10x better than 4, it will be clear that larger models are not facing diminishing marginal returns and all bets are off and any and every investment dollar will scramble in to larger and larger models. This will include the hyperscalers, financial institutions and likely even most state actors. We will reorganize society around building orders of magnitude larger models, requiring rethinking energy policy, capital, geopolitics, data, rare earth supply chains, advanced chip manufacturing etc etc etc. Waiting for OpenAI to reveal ChatGPT5’s leap is the only thing that matters in the world right now. Get out your 🍿 Which one are you betting on?

6

5 Comments

Anyone heard of AskJeeves.com ? Let me share an AI lesson from history… If you know AskJeeves, then you definitely belong to my pre-dot-com era pal list from 1997 (DM me!). This was the year when Yahoo was still a popular search engine, Google founders were still researching at Stanford labs. Jeeves was a popular AI avatar, mascot character and a Personal Assistant (Copilot) that answered queries similar to ChatGPT today with a hint of personalization (nearly 30 years ago) I vividly remember typing my queries in Netscape Navigator browser to AskJeeves in the late 90s. AJ.com was way ahead of the existing IT infrastructure those days (no GPUs for search engine) yet it fizzled as a product despite a butler mascot and loyally engaged customer base. AskJeeves.com, now known as Ask.com, has a rich history spanning nearly 3 decades. 1996: Ask Jeeves is founded by Garrett Gruener and David Warthen in Berkeley, California 1997: The search engine launches in beta in mid-April and fully launches in June 🚀 1999: Ask Jeeves conducts a successful IPO, capitalizing on the dot-com boom 2001: Ask Jeeves acquires Teoma for over $1.5M but then suffers a $425 million loss amid the dot-com bubble burst 2003: Ask Jeeves returns to profitability through an ad revenue deal with Google 2005: InterActiveCorp (IAC) acquires Ask Jeeves for $1.85 billion in July 2006: In February, Jeeves is removed from Ask Jeeves and the search engine is rebranded to Ask.com 2008: Ask acquires Lexico Publishing Group, which owns Dictionary.com, Thesaurus.com, and Reference.com 2009: Jeeves returns to the UK version of Ask.com in April, redesigned as a CGI character. 2010: Ask.com releases a mobile Q&A app for iPhone later that year. 2020: IAC/InterActiveCorp announces the retirement of the Ask.com brand as a standalone search engine. — What happened next and what were the results? The rebranding from Ask Jeeves to Ask.com in 2006 had significant impacts on the company: 1. Brand perception shift: The company moved away from just a "question-and-answer engine" to position itself as an "everyday search engine" competing with players like Google 2. Loss of character: While the rebranding simplified the brand, it sacrificed the company's unique character and personality that had been associated with the Jeeves butler 3. Mixed results: Despite efforts to modernize, Ask.com struggled to gain market share. By 2009, it accounted for only 1.67% of search queries and 0.42% of UK search engine visits 4. Reversal of strategy: In 2009, Ask.com revived the Jeeves character in the UK due to user demand and to bring "warmth and humanity" back to the search experience The rebranding ultimately had mixed results, leading to multiple strategy shifts as the company tried to find its place in the competitive search engine market. OpenAI.com and AskJeeves are eerily similar in their evolution and approach but hopefully, each can rewrite their concluding chapters in AI history books.

16

2 Comments

People often ask me, “What is DevRev, and what’s the vision behind it?” 🤔 Who better to explain it than our co-founder, Dheeraj Pandey? In this post, he shares the principles and innovative tech driving DevRev’s mission to connect makers (Dev) with customers (Rev). If you’re a startup, why not experience this vision in action? 🚀 Start your journey with https://devrev.ai/startups and focus on what truly matters! #DevRev #Startups #Innovation #GenAI

16

1 Comment

Most (maybe all?) LLM performance benchmarks like Artificial Analysis go in depth on *online* inference. *Batch* inference seems simpler since almost all companies run some form of embarrassingly parallel workloads. But batch inference is different from other map-reduce style workloads. The biggest differences are the use of GPUs and often mixed CPU + GPU compute. Today, tons of energy is going into improving systems for batch inference, and batch inference is just as important as online inference (companies have accumulated tons of data over the years, and if you want to use AI to extract insights or make decisions based on that data, you will need to process that data in a batch fashion). A bunch of companies are speaking at #RaySummit on their best practices for batch inference: Apple: "petabyte scale embedding generation" Roblox: "efficient batch inference at Roblox" Uber: "large scale generative AI batch prediction with Ray and vLLM" eBay: "pioneering next-gen AI platform" Handshake: "job description parsing with batch LLM inference" https://lnkd.in/deWkzHEC

260

4 Comments

Hiring full-time NLP engineer for fast-growing and funded generative AI startup! At Arrowhead, we’re becoming the AI partner for enterprises’ customer support and sales teams. We are working with enterprises globally and are targeting close to $1 mn in ARR by end of the year. We’re building proprietary in-house models that will create 100x value compared to what we can do today. At Arrowhead, you’ll experience a culture where:  🚀 You will have ownership and empowerment to take decisions, even as an intern 🚀 You will see what you’re building directly in the hands of users, and be able to iterate on a weekly basis based on user feedback 🚀 You will be challenged and pushed to do some of the best work you’ve ever done! If you are interested, please fill out the google form in the comments below 👇 #hiring #NLP #startuphiring

139

11 Comments

Customers do not care about LLMs beating other LLMs by a few points on public leaderboards.  𝐖𝐡𝐚𝐭 𝐦𝐚𝐭𝐭𝐞𝐫𝐬 𝐦𝐨𝐫𝐞 𝐭𝐨 𝐞𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐜𝐮𝐬𝐭𝐨𝐦𝐞𝐫𝐬 𝐚𝐫𝐞: ✅ 1. Accuracy : how accurate it is on my use cases? 🛂 2. Customizability: how easily can it be customized for my use cases? 💰 3. Cost: how cheaply can it be run as volume goes up? ⏩ 4. Speed: how fast it is in practice (think of inference speed)? 👌 5. Reliability: can it continue to produce high quality response in live env? 🤝 6. Compliance: can it meet all my regulatory and trust guidelines? ⛔ 7. Security and privacy: can it be trusted with sensitive data? ✚ 8. Integration: how easily it fit into my existing tech infra stack? 🔧 9. Easy of use: can my team evaluate and monitor and improve it to reduce barrier to adoption? 😓 10. Difficulty to replace: how difficult it is to replace if I have a better alternative? #b2b #ai #adoption #business #value

34

2 Comments

Loved all the thoughtful audience interaction at my "Deep Learning with PyTorch and TensorFlow" workshop at ODSC East yesterday. Halfway through, today's session on "Generative A.I. with LLMs" has been a blast so far too. Hope to meet you at Open Data Science Conference (ODSC) East in Boston this week if I haven't already! #datascience #odsc #odsceast #machinelearning #ai #generativeai

86

5 Comments

Haters will say it's fake but this is an actual image of Sundar Pichai showing up to this week's Q2 earnings call. (Stock up ~20% YTD) The wins: 1. Record Cloud Revenues: Google Cloud made over $10 billion in quarterly revenues for the first time, with $1 billion quarterly operating profit. This marks a 29% year-on-year increase in cloud revenues. 2. AI Innovation: More than 2 million developers are using Google's AI tools, contributing billions in AI-related revenues. The Gemini AI model family, used in products like Search and Workspace, has enhanced capabilities with 2 million tokens, the longest context window of any large-scale foundation model. 3. YouTube Growth: YouTube ads revenue grew by 13% year-on-year, reaching $8.7 billion, driven by strong brand advertising and a significant rise in YouTube Shorts and Connected TV views. YouTube remains the No. 1 streaming platform in the U.S. 4. Search Momentum: Google's search capabilities, enhanced by AI Overviews, saw increased engagement, particularly among younger users aged 18-24. Search and other advertising revenues grew by 14%, hitting $48.5 billion, with retail and financial services leading the growth. 5. Tech Infrastructure Investments: Google is heavily investing in its AI and cloud infrastructure. The launch of the sixth-generation TPU, Trillium, offers a near 5x increase in peak compute performance and 67% more energy efficiency compared to previous versions. Google also announced new data centers and cloud regions in Malaysia, Iowa, Virginia, and Ohio. 6. Subscription Strength: YouTube saw a 14% increase in subscription, platforms, and devices revenues, totaling $9.3 billion. This growth was driven by YouTube TV and YouTube Music Premium, along with Google One. The challenges: 1. Network Revenues Decline: Network advertising revenues fell by 5% year-on-year. 2. Increased Expenses: Total cost of revenues increased by 11% to $35.5 billion, driven by higher content acquisition costs, depreciation, and the impact of the Canadian Digital Services Tax. 3. Operating Expenses: Operating expenses rose by 5% to $21.8 billion, with significant increases in R&D costs primarily due to compensation and depreciation. 4. Real Estate and Technical Infrastructure Costs: Ongoing strategic investments in technical infrastructure and real estate are adding to expenses. 5. Subscription Growth Deceleration: Although subscription revenues grew, the year-on-year growth rate decelerated due to the anniversary of a YouTube TV price increase, suggesting that sustaining high growth rates in this segment may be challenging. 6. Cloud Profitability and Seasonal Patterns: Despite reaching a $1 billion quarterly operating profit, Google Cloud's margins are expected to follow a seasonal pattern with potential declines in Q3 due to increased expenses associated with technical infrastructure investments.

7

Enjoyable. As a Product Manager, you should obsess over that concept.   Yesterday, I chatted with my friend Abhishek Sinha, a senior product leader at NVIDIA, and ex Cohere and Amazon Web Services (AWS).   In front of 40+ Surge founders, we went deep into all things product management. Some insights worth sharing:   👉 “What is a developer?” - it’s important to go deep and specific on your ICP. For example, you can’t just say that your ICP is a developer. What is that? Frontend? Back-end? Data? In SME? Or Enterprise? There are 20+ different type of developers, each with different needs and motivations. Be specific. 👉 All about motivations- definition of personas and segments should not be done based on superficial demographics. Rather, understand the WHY? Understand the motivations, deeply. Why are they using your product? What drives them? Group them based on the WHY. 👉 60% rule. Product Management is not just about drafting PRD’s and project managing build & launch. It’s all about customers. Working backwards. Those motivations? You can only understand them when you spend at least 60% of you time speaking with customers. Anything less, and you’ll miss stuff. 👉 Storytelling is everything - a product can only succeed when the narrative is strong. Communication is key and product marketing should be joined at the hip and part of all key customer conversations. Again, work backwards from the customer and develop clear messaging and value prop, optimized for your different channels.   Which brings us back to enjoyable. That value proposition? It can only be about 3 things: faster, cheaper, more enjoyable. That last one is not the same as ‘easy-to-use’. Some products are not easy, but so unique, so powerful, so rich that they can be hugely enjoyable.   So aim for that. Aim for enjoyable. cc. Peak XV Partners Anuj Sahai Muharrem Derinkök Jon Woods Pieter-Paul Walraven Jascha Zittel Elliott Gibb Nellie Wartoft James Han Carla Penn-Kahn Nitish Tiwari Ajain Vivek

149

13 Comments

Check this out 🚀 As aptly mentioned in the comment section - Exceptional founders, amazing product!

3

I believe that the democratization of LLMs across all languages is a massive task ahead of us, especially in a country like India, where we have 780 languages spoken within the country. I invite you all to test out a new GPT I have quickly configured, called GanitGPT, which helps learn Math in your regional language. However, because I know only Hindi, Maithili and Bangla, it was hard for me to provide examples in other languages, I just want to know how the ChatGPT model, optimized by a few prompts and a few math texts, performs in languages part from the ones mentioned above. Try this out: https://lnkd.in/dtXnx77T #chatgpt #math #indianlanguages #largelanguagemodels

19

4 Comments

AI Self Healing is live AQUMEN LABS. Over the past month, the team has been working hard to reduce costs incurred by AI during automation. AI self-healing helped us in : 1. Reducing test writing and automation by 90% - We wrote a test for login and completing an FTUE of a game in just 2 minutes [Faster than other low code tools] and not to mention the Game is in UNITY [Yes we support Unity :) ]. 2. Reduce the Maintenance of the test script by 80%. The test will be healed when you select the Heal checkbox on your test step. 3. We have successfully implemented a Hybrid approach, where our framework is intelligent enough to decide whether to use AI, which helped us in reducing the cost. For cost reference: MultiOn [an agentic AI] uses 0.04$ per step and AQUMEN LABS uses 0.03$ per step executed and that too in an intelligent way :) This is purely a Vision-Based Multi-Model approach we do not use any locators to heal. If you and your team are exploring an AI approach in testing, we would be happy to showcase it. Schedule a call: https://lnkd.in/gmu62wzV Note: Used Multion Example as it is one of the best agentic AI tools that we follow and take inspiration from. Multion and AQUMEN LABS are two totally different tools for solving different use cases.

5

🚨 PSA 🚨 YOU DO NOT NEED TO FINE-TUNE TO GET YOUR MODEL TO OUTPUT STRUCTURED TEXT Outputting structured text is essential for most enterprise AI use cases - as failure to adhere often leads to breaking of downstream pipelines. A good inference server should give you the ability to turn on 'JSON Mode'. DM me if you want to learn more about how to get models to output structured text with 100% reliability. (Post inspired by Snowflake's fine-tuning demonstration)

49

11 Comments

There's a common belief that scaling 3 dimensions - model size, training compute and size of the dataset will lead us to AGI. But, this is an oversimplified view of reality. Read an interesting piece on #AISnakeOil from Sayash Kapoor and Arvind Narayanan that critiques the widespread belief and provides a wonderful explanation. While scaling has improved model predictability, the real value lies in unexpected capabilities that appear as models grow. They term it as - the "Emergent Abilities" of the model. Undoubtedly, until now scaling up has led to remarkable improvements in the predictability of responses. While this is interesting from a technical standpoint, it might not necessarily be what end users care about. Maybe what really matters is the unexpected capabilities that models acquire as they grow in size. Another misconception is the role of synthetic data. While it’s often considered as a solution to the data scarcity problem, it’s primarily used to fill specific gaps or enhance domain-specific tasks, not to increase the overall volume of training data. In summary, the future of AI and LLM development isn’t about endless scaling. It’s about smarter, more efficient use of resources to unlock the best outcomes. Apologies for the long post but I found it super interesting. What are your thoughts? You can read the complete article here 👇

18

2 Comments

I believe a solution to this is currently being developed.

5

Brilliant write-up of another MOVEit SFTP server bug. The essential idea behind this multi-pronged exploit: a) Upload a public key to the target SFTP server to a known location. How? Simple: Exploit server logging :-) The researchers found that the server logged copiously. So they found an unauthenticated endpoint meant for guest users and supplied the key as payload. The server dutifully logged the key contents into the log file. Now because key files MUST begin with the ----- BEGIN PUBLIC KEY ----- text, the researchers needed to find a key format that the SFTP server accepted that would accept any file and strip out the contents until it found a valid key. They found it in the PPK format (Putty keys). So they converted their key to this format, then invoked the endpoint with the PPK key as payload and voila their public key is now resident in a log file on the target server. b) Now authenticate as a user on the target server except that you provide the path to the log file containing the public key in the previous step. Sign this request with the same key.  c) This authentication will succeed despite attacker not knowing the keys of the user being impersonated. This occurs because of a bug in how the authentication is handled and the interplay with a lower level SSH library being used. The researchers seem to imply that code reviews may have caught this bug. But it's not so simple. Fuzz testing would indeed have caught this and certainly code coverage driven testing because an error in one code path was exploited to trigger a bug in another. A final point: It's hard to see why the entire payload needs to be logged, or even why sensitive data like keys (even if public keys) are not masked.  Read this excellent write-up - link in comments - and get ideas on the ways attackers think - as better ways to secure your own code.

2

1 Comment

It took us < 30 mins to integrate Llama3 to our health AI stack. It takes our customers < 3 mins. Within a week Llama3 went live with all the 101 GenAI healthtech customers. We’ve curated and integrated 15+ models over last 7+ months to enhance healthcare workflows. Don’t spend weeks or months integrating with just one model; instead have a “team of models” power your “network of copilots”. Great post by Prashanth Aditya Susarla about how to build a high performance AI product with team of LLMs.

16

Totally want Indian origin folks to be celebrated and interviewed. And hearing Aravind Stinivas is always a learning experience. I enjoyed his eloquence in explaining concepts on the Lex podcast. A very erudite and accessible speaker of some of the most sophisticated technology development happening in AI. But his lack of green card has nothing to do with it. It absolutely is something American government needs to reflect on and future immigrants. But an interview of Srinivas has nothing to do with a green card if the title talks about his product is disrupting Google search. It’s not like Pichai only started delivering at Google after he got his green card. Yeah, the long time it takes probably coincides with the arc of great achievements that folks like Nadella and Pichai have achieved. Those long arcs are woefully the aspect of the US Immigration system. But has no material bearing on folks like Srinivas. Sure, for someone like me, I may excuse myself from taking such bold bets or on entrepreneurship due to lack of green card. But there are a ton more folks like me vs folks like Srinivas. Folks like me may not have the ambition of disrupting a Google (hint- I surely don’t). Folks like me may not even know how to disrupt any existing product or business out there. And yes, I may make an excuse for all these gaps by saying that I don’t have a green card. And I do recall saying things like this when I lived in California. But there were a ton of nodding heads to what I said. I guarantee you, folks like Aravind Srinivas don’t hang out listening to excuses!! So, let’s not bring the notion of such thoughts of green card excuse in any near vicinity of folks who are meaningfully treading where angels fear to tread!! And they aren’t fools rushing in!! They are finally angels treading and stomping their way to the pearly gates! No green card needed to storm the citadel of Google!!

20

2 Comments