Feiya Shao

#PET #OWNERSHIP? Reminder of proxy variables Looking at map of pet dog ownership tells… … easily mistaken story. The clear geographical delineation in graph (matching the cold war Iron curtain line) can easily sway you into cultural root cause. 🤦🏻‍♂️ And that would be a great example of the correlation fellacy. Because 25 years after the Iron curtain is down is 2-3 generations of the dogs, long enough time for the political factors to wear off, if they ever had effect. 📖 The real learning of this graph is that there might be hidden factor that is not obvious and still it has profound effect on the actual “target variable”. 🐕 In our dog situation the underlying hidden root cause for the pet ownership is the share of agriculture on total labor market 1.5-2 generations back. Because if you had a farm, you were more likely to get used to living with animals or even needed some (shepherd dogs anybody?). The 2 generations lag is important, because you don’t make pet purchase decisions until you live in your own household and that is roughly, when you start your family (hence when you pass on the generation baton). 🔗 This phenomena is also known as #proxy #parameter and has significant impact also (in positive means) in #MachineLearning. Because proxy parameters are strong predictors, but are insights that markets tend to oversee. So if you know them, proxy parameters give you edge. 💡 Now you know the phenomenon I have a #puzzle for you: It is proven by data in many insurance companies (I worked on one of such studies) that #Zodiac sign is very strong predictor of the #car #accident rate. Yes, really true! Can you guess what it is proxy parameter of?

3

Come and build a smart shopping assistant designed to answer customer questions, guide product discovery, and tailor search results. Combine the power of AlloyDB, Gemini (Large Language Model) and Vertex AI Agent Builder to quickly bootstrap an intelligent conversational assistant.

7

📊 The Power of High-Quality, Diverse, Real-World Data Fine-Tuning LLMs 🌍 When it comes to training Large Language Models (LLMs), the mantra isn’t just “more data”—it’s better data. While having more data can be beneficial, it’s the quality, diversity, and real-world relevance of the data that truly drive impactful results. Here’s why: 1️⃣ Higher Quality: clean, well-curated data improves model accuracy and reduces noise, making the model more reliable and stable. Poor-quality data often leads to inconsistent responses and increased hallucinations. Unbiased labeling consistency is key. 2️⃣ Diversity: it ensures the model can generalize across different contexts and user scenarios. This makes the model more robust, safe and inclusive, which is especially important in NLP for avoiding bias and ensuring representation. 3️⃣ Real-World Datasets: training on real datasets helps models perform better in real-world applications, as it learns from actual patterns and usage scenarios, making the model more practical and effective. For LLMs, there’s a general playbook to prepare data effectively: (1) Collect Instruction-Response Pairs: gather examples of how tasks should be answered, covering various domains and complexity levels. (2) Concatenate Pairs: organize the data into structured sequences, allowing the model to learn from end-to-end interactions. (3) Tokenize: process the data into tokens, using padding and/ or truncation as needed to maintain uniform length. I particularly love 🤗 transformers library's AutoTokenizer to identify which tokenizer to use on every model. (4) Split into Train/Dev/Test Sets: partition your data into training, validation, and testing sets to ensure proper model evaluation and tuning. Ultimately, data quality prevails over quantity, specially as we live in the LLMs era when models have been pre-trained with almost all the public data available in the 🌎 . A well-prepared dataset tailored to your application will make all the difference in the performance and reliability of your model. #AI #MachineLearning #DataQuality #LLM #NLP #ModelTraining #DataPreparation

5

What’s the Real Context Size of Your Long-Context LLMs? Picking up from my last post (https://lnkd.in/gYtgs_DT), where I talked about the pros and cons of long-context LLMs in Retrieval-Augmented Generation. This NVIDIA research takes a closer look at how these models perform at their limits, with insightful findings that reveal some non-obvious challenges. The authors tested 17 language models, including well-known ones like Gemini 1.5, GPT-4, and Llama 3.1. Turns out, despite many models claim to handle huge contexts, very few actually keep up their performance as context length grows. This might come as a surprise if you're familiar with traditional benchmarks like "needle-in-a-haystack" and passkey retrieval, where most models achieve near-perfect accuracy. While those tests are useful, the new RULER benchmark pushes LLMs with more complex tasks, such as multi-hop tracing and aggregating information across long contexts, ultimately revealing their true limitations and areas for improvement. As language models grow in size and context capacity, it's easy to get excited about their new capabilities and forget the trade-offs. Just because a model can handle massive contexts doesn’t mean more is always better — often, keeping it small and simple still wins the day. I'll drop the link to the paper in the comments. #MachineLearning #ArtificialIntelligence #AI #LLM #GenAI #NeuralNetworks

57

3 Comments

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

Meta's Llama 3: A watershed moment for Open-Source LLMs? Exciting news in the tech world as Meta's launch of Llama 3 last week (https://lnkd.in/gk7VhheQ) is redefining the open source LLM landscape! 🚀   While 2023 saw remarkable strides with Llama 2, Falcon 180B, and Mistral 7B, Llama 3 is setting new standards with its unparalleled raw performance. The larger 70B version is now only second to OpenAI’s GPT-4 Turbo on the LMSYS leaderboard for English prompts (https://lnkd.in/gce2yKhy), and the smaller 8B model outperforms GPT-3.5, which powers the free version of ChatGPT. 🏆   Users are raving about Llama 3's faster inference compared to GPT, making it a game changer in the realm of super-fast image generation. Its integration with Meta products allows for real-time modifications with prompts, vastly enhancing user experience. Since it is open source, folks on X have even shown how incredible it is when paired with Groq!🌐   What's fascinating is that Llama 3 used ~75x more data (15T compared to ~200B) than the Chinchilla paper's benchmark (https://lnkd.in/gzv5H3FN), and Meta claims it still wasn't converging. This underscores the vast potential of data in enhancing models with sub-100B parameters. Yes, the training costs are higher, but the lower parameter size reduces inference costs, paving the way for running LLMs on local devices in the future! 💡   The only downside? A context length of just 8k when 128k is becoming the norm across top models. But hey, every innovation has room for improvement, right? And they have indicated that this is bound to increase in the coming months as well!   Kudos to Meta for pushing the boundaries of open-source development with Llama 3. Its impressive performance, combined with Meta's product integration, makes this a development worth watching! Do you think the 400B+ model (summer launch expected) would beat OpenAI's GPT 4 Turbo and maybe even be as good as GPT 5?🎉   #Meta #Llama3 #OpenSource #AI #LLM #TechInnovation #MachineLearning #GenAI #GPT4 #OpenAI

25

5 Comments

Thrilled to see MLCommons launch version 1.0 of the AI-Luminate benchmark, paving the way for clear, apples-to-apples comparisons on model safety. Developing this benchmark required a foundational step: creating a unified taxonomy of hazards so that we can all be speaking the same language and using that taxonomy as the basis for model safety testing. We wanted the hazards to be mutually exclusive, easily understood by users of all technical backgrounds, and comprehensive enough to cover the spectrum of potential risks in text generation --- and, of course, with this being version 1.0, we expect continued iteration and evolution of this taxonomy (any/all feedback welcome!). It was particularly great to get to work on that harms taxonomy with partners from across academia, industry and civil society. Particular shoutout to Eleonora Presani, Brian Fuller, Carlos Ignacio Gutierrez, PhD and Peter Mattson among many many other MLCommons members and collaborators that made this happen. https://lnkd.in/eD4Pk_sC

36

🚀 Does Size Always Matter? 🧐 When it comes to Large Language Models (LLMs), the answer isn’t always straightforward. While larger models often top leaderboards with impressive average benchmark scores, real-world applications tell a different story. A few considerations: • 📊 Leaderboards favor larger models, but these models might not always be the best for specific tasks. • 🎯 For business applications, we often use LLMs for single tasks or a narrow subset of tasks, which align with specific benchmarks, not the whole portfolio. • 💡 A great example is Bielik 2 - a Polish LLM with just 11B parameters, performing well in SpeakLeash.org Arena, where it ranks between LLaMA 3.1 405B and 70B (this ranking reflects how well the responses of different models meet the subjective expectations of Polish users). • 💸 This becomes especially important when the models under consideration differ significantly in terms of cost. Key Takeaway: When choosing an LLM, prioritize benchmarks or rankings that closely match your use case over general leaderboards, or even create your own benchmarks closely aligned with your needs. #AI #MachineLearning #LLM #BusinessInsights #TechTrends #AIModels

52

5 Comments

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

147

1 Comment

The best learning during my time at Amazon was improving my writing. Here is my list of how to edit your writing to improve it and how to get an idea down on paper in the first place. TL;DR craft compelling documents by anticipating reader needs, addressing potential questions, and clearly articulating the desired outcome. This will take time and a lot of editing. Make your writing better, editing tips: A. Use data and logic to lead the reader to a conclusion. Using adverb and adjective is less effective. B. Eliminate weasel words C. Use readable scores to get data on how readable your doc is now and improve it D. Use read-a-loud extension or read it out loud yourself E. Edit words that don't add value. Look for repetitive language. Don't tax your reader with big words How to crafting an idea in writing: 1. Start with bullet list of what your thinking about 2. Start by writing with limited edits. Sleep on it and then edit like crazy. 3. Know your reader and write for them. 4. Write down FAQ to the end of a doc that you think your reader will have. If you can, you should answer these in your writing but it's not always possible.  5. Take ownership not ask for ownership. If you are asking for ownership or asking the reader to force someone to do something you are doing it wrong.    6. Ask why 5 times on customer facing questions to get to the detail why this is important 7. Think in 3s. How would you explain this with a 3 word (The Title), 3 sentences (TLDR), 3 paragraphs (a 1 page page overview), 3 pages (longer form with more detail). 8. Bonus: add some long term thinking topics to your doc. If the reader is excited about what you are writing about, give them more and think big about the future. 9. Close with what decision you need help from the reader 10. Finally writing a couple page document should take a week(s) not days or hours to complete.

78

7 Comments

What people leaders can learn from Large Language Models (LLMs) Because LLMs are trained from human data, it's unsurprising that their behavior mirrors that of their human counterparts. It's worth taking a closer look at human teams through the lens of what we've learned about LLM performance. While it wasn't the authors' goal, this paper highlights many LLM issues and recommendations that parallel what is needed for human business teams. Here are the key takeaways from section 5. - Assigning tasks to individuals based on their specialties improves performance. - A clearly identified leader and a comprehensive feedback system will keep the team on track. - Clear role definition ensures that individuals stay focused and reduces mistakes. - Planning and task decomposition improve decision-making and execution. As a final note, the paper mentions the use of a rotating leader to improve the overall performance of the multi-agent response. Applied to human teams, this is a novel approach with great potential. Leaders who stay connected to technological developments would inject that fresh knowledge into their teams.  What team wouldn't benefit from periodically cycling through people leaders who have kept their skills sharp and relevant? https://lnkd.in/gXUp88th

22

So much fun talking about the power of LLMs with Amazon Bedrock and Anthropic Claude. In summary, the ever-evolving landscape of GenAI, it's becoming clear that there won't be a single LLM (Large Language Model) to rule them all. The rise of application-specific LLMs heralds a new era, where various models will carve out niches to excel in. This diversity opens up a wide playing field for innovation and tailored solutions. When it comes to deploying AI models, striking the right balance in evaluation is paramount. We must weigh factors like cost, speed, and precision to ensure optimal performance and resource utilization. Finding this equilibrium ensures that our GenAI solutions are not just efficient but also effective in real-world scenarios. Anthropic Claude emerges as a frontrunner in the realm of LLMs, distinguished by its exceptional capabilities and groundbreaking advancements. Boasting unparalleled scale, efficiency, and adaptability, Anthropic Claude represents a pinnacle of GenAI innovation. Its ability to seamlessly integrate with Amazon Bedrock amplifies its prowess, providing users with an unparalleled ecosystem for knowledge integration and generation. With state-of-the-art architecture and a commitment to pushing the boundaries of what's possible, Anthropic Claude stands as a testament to the transformative potential of GenAI in shaping the future of technology and beyond. Prompt engineering holds the key to maximizing the potential of applications. Crafting prompts that consider persona, context, tags, and positioning can significantly enhance the model's ability to comprehend and generate relevant responses. This strategic approach empowers AI systems to better serve their intended purposes. In the realm of GenAI architectures, a robust knowledge base is essential. Amazon Bedrock stands out as a key component for architectures like RAG (Retrieval-Augmented Generation), providing a solid foundation to fuel intelligent decision-making and content generation. As we march towards AI integration, bridging the gap with legacy applications is crucial. Agents equipped with open APIs offer a seamless way to facilitate communication and interoperability, ensuring a smooth transition into the AI-driven future. Let's continue to explore, innovate, and collaborate as we navigate the exciting frontier of GenAI together! #aws #LanguageModels #Innovation #TechTrends #AIIntegration #webdevcon #claude #bedrock

68

(I've spent more time than I'm willing to admit working on this post. Hope you get as much satisfaction from reading it as I get from publishing 😅) At Motif, the problem we've heard from almost every potential customer is that they want to understand what drives the successful and unsuccessful outcomes in their product. To reliably answer these causal discovery questions, we developed an entirely new approach that combined ideas from 1) causal inference and 2) sequence models currently powering the generative AI systems. The result is a methodology for automated causal inference that scales to thousands of cause-effect hypotheses, allowing users to efficiently find and prioritize the mechanisms for developing and prioritizing product improvements. In our early case studies, this approach is incredibly powerful and I am excited to see what we learn as we scale this out to our customers.

417

17 Comments

I wrote an article with David Yuan and Diego Z. from Tidemark, where we dove into how AI is shaking up the data game. For years, the big players—those with massive datasets—had an edge because their data acted like “gravity,” pulling customers in and keeping them there. But AI is changing that. Now, with tools to integrate and make sense of data across systems, startups can compete without needing to play by the big guys’ rules. It’s a new era of agility where data orchestration may just be the next big advantage. Exciting times for those rethinking their data strategy! https://lnkd.in/eqGDACYW #AI #DataStrategy #Innovation #Partnership

24

2 Comments

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

Ensemble of smaller LLMs (PoLL) is less biased, faster, 7x cheaper when evaluating LLMs performance. Works great on QA & Arena-hard evals arxiv.org/abs/2404.18796

9

I have read several posts about LLMs picking not-so-random numbers when asked to pick a random number. This study is comprehensive, and confirms the observation. https://lnkd.in/gU-4KGkD Of course, this was not exactly the right question to ask. LLMs' picks of random numbers are biased towards the training data, and that adversely affects the randomness. The more important question; what are the questions that we should avoid relying on generic LLMs to answer correctly?

4