Akshaj Jain

LinkedIn team (Juan Pablo Bottaro Karthik Ramgopal) shared a helpful deep dive into their deployment of an LLM-powered job search assistant. Highly recommend this read! Few highlights:  💡 Relatively easy to setup RAG-based pipeline. Achieved 80% of the basic experience in 1 month, then 4 more months to reach 95% mark. 💡 Evaluating the quality of application outputs is hard. I hear this from many enterprises developing LLM applications. (check out Okareo by Matthew Wyman Boris Selitser) 💡 Balancing latency with quality is important. While certain techniques like Chain of Thought could improve accuracy and reduce hallucinations, it will increase response times, which can affect user experience. https://lnkd.in/gtGGwf5P #largelanguagemodels #RAG #EBR #LLM #ml #ai #hallucinations #evaluations #

4

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

6

1 Comment

Quentin H. and Yanwei (Wayne) Zhang on the Faire Data team recently did some great work fine-tuning Llama3 model to enhance search relevance. Here are some key insights from the work: • While prompt engineering is valuable, LLM fine-tuning is essential for a domain specific task like understanding semantic search relevance. • Fine-tuned open-source LLMs like Llama3 can deliver impressive performance, scalability, and cost-efficiency compared to using proprietary models. • Effective application of a fine-tuned LLM demands a well-defined problem, robust, high-quality labeled data, and ongoing model refinement. Ground-truth labeled data is crucial for maximizing model performance. • Fine-tuned Llama3-8b achieves performance on par with the larger Llama2–13b model. Read the blog post to dive deeper into these findings! https://lnkd.in/ecP_3E7S We are also hiring (https://lnkd.in/dHfRZcPi) #LLM #AI #Finetuning #LLAMA3 #SearchRelevance #Hiring

93

3 Comments

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments

Parametric design is about to change forever. Current approaches to code generation for design are limited because they don't learn from code execution feedback during training, making them static by nature. This is where the RLEF framework comes in. RLEF enhances code language models through a comprehensive training process: → Pretraining: Teaching the neural network to predict the next token by learning from vast amounts of text data. → Fine-tuning: Instructing the pre-trained model to follow specific prompts and respond accurately. → Post-training: Aligning the model with human preferences using reinforcement learning, specifically tailored for code generation with RLEF. RLEF enables these models to learn from execution feedback, allowing them to produce not just syntactically correct code, but functional code that meets intended outcomes. By understanding and iteratively improving based on execution results, RLEF transforms static code generation into a dynamic and intelligent process. Why does this matter for parametric design? In our field, design parameters aren't just variables, they're the fundamental building blocks of innovation. Automating parametric design requires code that not only adapts but also intelligently evolves with each iteration. Traditional AI code generators often leave us with code that needs extensive tweaking. But with frameworks like RLEF, we can envision an AI that understands and learns from execution. It doesn't stop at code generation; it evaluates the output and learns from successes and failures. Through reinforcement learning, it continuously improves to better meet design objectives. Less time debugging means more time innovating. Imagine AI models that can automatically adjust design parameters based on real-time feedback, optimize complex designs without constant human intervention, and accelerate development cycles by learning from each execution. This is what I call spoken design ;) The potential of RLEF isn't just theoretical; it's a tangible step toward more intelligent, responsive design systems. Credits to Facebook AI research team for their amazing work. Exciting times ahead for those passionate about the fusion of AI and design. Link to the paper: https://lnkd.in/eFgAW8qC

27

Last week, Apple researchers released a paper that was quickly embraced by AI skeptics as evidence that large language models can't reason. The study highlighted how seemingly minor modifications to benchmark problems, such as changing numbers or adding irrelevant details, can cause performance drops. The findings suggest that LLMs rely heavily on pattern matching rather than genuine logical reasoning. However, these apparent limitations don't tell the whole story. Microsoft’s Orca 2 paper provides a contrasting perspective by demonstrating that LLMs can be taught to reason through structured learning and task-specific strategies. Orca 2's approach of progressive learning—using few-shot, zero-shot, and step-by-step reasoning techniques—reveals that, with the right training signals, LLMs can develop specialized reasoning capabilities. What the Apple study highlights as brittleness is, to some extent, a consequence of the current state of general-purpose models. These models aren’t inherently designed for domain-specific tasks, such as mathematical reasoning, and they struggle when encountering variations they haven't been trained to handle. In contrast, Orca 2 demonstrates that teaching models explicitly how to think through a task—whether through explanation-based learning or multi-step reasoning—can meaningfully enhance their problem-solving abilities. LLMs, like any technology, must be applied with intention. While general-purpose models may struggle with some tasks out of the box, fine-tuning and careful guidance enable them to thrive in specific domains. The path forward for AI reasoning isn't to dismiss current models' limitations but to acknowledge the value of teaching them how to reason within their scope—as Orca 2 has shown. This iterative process of learning, adaptation, and fine-tuning is what will drive LLMs toward greater reliability and utility across diverse applications. The conversation shouldn't just focus on what LLMs can't do—it should also consider what they can learn to do with the right strategies. Listen to dueling viewpoints of Apple vs. Orca papers in the following Google NotebookLM: https://lnkd.in/gKP4kMV6

129

94 Comments

I'm a computer science professor at UC Berkeley. Tech jobs are drying up and graduates are no longer guaranteed a role. https://buff.ly/3Y9jVRf Key Takeaway The traditional path to tech employment through earning a degree is becoming less reliable, with factors like AI, outsourcing, and a shrinking job market requiring a re-evaluation of employment strategies and support systems, such as universal basic income. Summary -UC Berkeley professor James O'Brien notes tech graduates are struggling to secure jobs due to a shrinking market. -Outsourcing and AI are reducing entry-level tech positions. -O'Brien suggests considering universal basic income to support workers as AI continues to reduce job availability.

1

Vinod's thoughts on AI: Key Takeaway 1: The impact of AI on various industries is expected to be highly deflationary. As AI automates many tasks and reduces labor costs, prices for goods and services are likely to decrease significantly across sectors like healthcare, content creation, and professional services. Key Takeaway 2: There are still opportunities for AI startups, particularly in niche applications and specialized domains. While large language models may become commoditized, there is potential for domain-specific AI solutions that integrate workflow knowledge and industry expertise. Key Takeaway 3: The cost of using AI models is expected to decrease dramatically in the near future due to competition between major players. This will likely make AI more accessible and ubiquitous, though training costs remain high. Key Takeaway 4: Apple's recent AI announcements demonstrate the company's ability to integrate technology, platforms, and products. This positions Apple well to bring small on-device AI models to consumers and developers at scale. Key Takeaway 5: The upcoming U.S. presidential election has significant implications for the tech industry and AI development. There are differing views in Silicon Valley on regulation and policy approaches, with some favoring less regulation while others are concerned about ethical issues. https://lnkd.in/gG9hSuUy

12

I wrote last week about why working on a concrete startup or project idea — meaning a specific product envisioned in enough detail that we can build it for a specific target user — lets you go faster. In this letter, I’d like to share some best practices for identifying promising ideas. AI Fund works with many corporate partners to identify ideas, often involving applications of AI to the company’s domain. Because AI is applicable to numerous sectors such as retail, energy, logistics and finance, I’ve found working with domain experts who know these areas well immensely helpful for identifying what applications are worth building in these areas. Our brainstorming process starts with recommending that a large number of key contributors at our partner corporation (at least 10 but sometimes well over 100) gain a non-technical, business-level understanding of AI and what it can and can’t do. Taking DeepLearning.AI’s “Generative AI for Everyone” course is a popular option, after which a company is well positioned to assign a small team to coordinate a brainstorming process, followed by a prioritization exercise to pick what to work on. The brainstorming process can be supported by a task-based analysis of jobs in which we decompose employees’ jobs into tasks to identify which ones might be automated or augmented using AI. Here are some best practices for these activities: (i) Trust the domain expert’s gut. A domain expert who has worked for years in a particular sector will have well honed instincts that let them make leaps that would take a non-expert weeks of research. Let’s say we’re working with a financial services expert and have developed a vague idea (“build a chatbot for financial advice”). To turn this into a concrete idea, we might need to answer questions such as what areas of finance to target (should we focus on budgeting, investing, or insurance?) and what types of user to serve (fresh graduates, mortgage applicants, new parents, or retirees?) Even a domain expert who has spent years giving financial advice might not know the best answer, but a choice made via their gut gives a quick way to get to one plausible concrete idea. Of course, if market-research data can be obtained quickly to support this decision, we should take advantage of it. But to avoid slowing down too much, we’ve found that experts’ gut reactions work well and are a quick way to make decisions. So, if I’m handed a non-concrete idea, I often ask a domain expert to use their gut — and nothing else — to quickly make decisions as needed to make the idea concrete. The resulting idea is only a starting point to be tweaked over time. If, in the discussion, the domain expert picks one option but seems very hesitant to disregard a different option, then we can also keep the second option as a back-up that we can quickly pivot to if the initial one no longer looks promising. [Reached LinkedIn length limit. Full text: https://lnkd.in/g4yFKXp2 ]

3,478

121 Comments

🎙️ Curious about what it really takes to thrive as a freelance AI consultant? 🔍 On Friday, October 25th, 9:30 AM - 11:30 AM EDT, I’ll be joining Jason Liu, an independent consultant with experience at Meta and Stitch Fix, for a live podcast recording of The Art of Freelance AI Consulting and Products. We’ll dig into real-world strategies for navigating the transition from freelancer to successful consultant, and explore the practical steps for building scalable AI systems that deliver real business impact. 💡 What we’ll be covering:  * How Jason went from $170/hour consulting to securing $60k upfront contracts and his approach to value-based pricing. * The technical shift from deterministic software to probabilistic AI systems, and how to guide teams through this transition. * A live role-play session where I’ll be the one getting coached by Jason on avoiding consulting pitfalls and building better client relationships. Should be fun! 🎭 Whether you’re a freelancer, consultant, or AI enthusiast, this episode will offer straightforward, actionable advice on both the technical and business aspects of consulting. 👉 Sign up now to join the live stream: https://lnkd.in/e2JTcRKT

20

4 Comments

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

This is very true: “Over the last 70 years, AI researchers have repeatedly made the mistake of trying to bake human knowledge into AI systems, only to be eventually outperformed by more general methods using brute force compute.” Take for example data manipulations. AI is good at generating answers but are they correct? The time it will take humans to verify a report created by AI would exceed the time to create it. Why? Humans have to understand the ligic behind the numbers. But AI does not offer a glimpse into such logic putting the human verifier into hell loop.

4

I recently read this paper about the tug-of-war between RAGs and their generative models' internal priors and ran a few short tests on what this could mean for us when using LLMs for fact extraction. Read below and let me know if you had a similar experience!

12

1 Comment

Large Language Models (LLMs) as super lossy compression. I have been thinking a lot about demystifying what LLMs do because lots of y'all's are very mystified. In the linked article, I try to reframe them for you, hoping it will help you understand what they good for and what they are not good for. Side note: I am still looking for my next adventure. If communicating complicated concepts so that regular people can grasp them is something you need, you should message me. I write code and stuff too and am usually fun to be around. #WrittenByMe

6

3 Comments

Who will get crushed by Claude LLM coming to iPhones? Will existing apps (and companies) get steamrolled by this? For those that don’t know Claude is a suite of 3 LLM models: Haiku, Sonnet and Opus. - Opus knocked ChatGPT off the leaderboard on capability when it was introduced. - It can hold a massive 160,000 words “enough for a user to paste in a weighty novel and ask follow-up questions” - Claude is multimodal so you will be able to take pictures or upload images from your iPhone and include that in your prompt Use cases * After a meeting, a business user could snap a photo of a whiteboard diagram and ask Claude to summarise the key points, making it easier to share and act upon important information. * Similarly, a consumer could take a picture of a plant they encounter on a hike and ask Claude to identify the species and provide more information about its characteristics and habitat. ** think the Picture This app ** What other uses do you see for this? https://lnkd.in/gdExfpx9

6

This article explains how (some) VCs take value and turn it into personal fortunes, taking money out of productive use for speculative markets. Personal note: I’m a (very little) VC, a former SME CFO and controller-accountant, and an MBA. I cap our equity asks at investment value + reasonable and patient interest. So that value stays IN the company and we aren’t building companies to sell the company, we’re building them to support founders and employees in creating real value and a good life for them and their customers. No less. Some notes on the article: The 1/10 «unicorns» that typical VCs keep and grow are typically the companies that didn’t need a VC in the first place. A direct investment would have been far more advantageous to the founders and the staff. VCs are mostly terrible at DEI, too. «According to State of European Tech 2022, 87% of all VC funding in Europe is still raised by men-only founding teams, while funding raised by women-only teams dropped from 3% to 1% from 2018, and only 1.4% of European unicorns are made up of entirely minority ethnic entrepreneurs.» From: https://lnkd.in/e7wadXmT https://lnkd.in/e-KP-F8M 📣👋 If you’re feeling like hey *I* don’t do business that way: good. Get in touch.

1

1 Comment