Andrew McGlathery

Deep in the Llama 3 paper, Meta tested how common prompt tactics affect results: Not at all. On page 31, they told the model: 👨‍⚕ You are an expert ✅ Choose the best answer And found ❌no❌ measurable difference in MMLU scores, across all model sizes. You don't need to tell the bot to be "helpful" either! Only give it a persona if you need it to act other than a smart expert. And then, just ask the question. This makes sense, because these models are post-trained on how they hope humans will use it - no "gotchyas" and secret hacks required!

5

3 Comments

What's not being good at interviewing costing you? "I just cannot seem to interview well. I have lost out on roles to people who used to report to me." I was speaking to my client—let's call him Steve. He was a smart, accomplished product leader who had led successful launches of products you have probably heard of. He launched multiple hit products, drove millions in revenue, and all that jazz. But Steven had a problem—a big one. Despite being a total PM rockstar, he still interviewed like he had seven years ago. When he was interviewed, the impression he left was "Great product manager, not leadership material yet." Ever felt like you're the most accomplished person in the room, but when it comes to interviews, you can't seem to convey that? That was Steve. Do you know who didn't choke on interviews? The PM's Steve personally trained and mentored. That's right - Steve's own pupils interviewed well rose above their sensei by having better storytelling and positioning, and the executive presence and confidence that got them further in an interview. His pupils had risen to levels above him. Can you imagine how emasculating and demoralizing that must have felt? To have your star pupils, whom you groomed from scratch, outshine and outearn you just because they could describe their achievements better? Steve had already missed out on many years of success, new roles, and millions of dollars because he was worse at interviewing. He felt the sting of watching others, who he knew he was more qualified than, climb the ranks and achieve the success he deserved. This wasn't just a financial loss but a blow to his self-esteem. He wanted his kids to look up to him and be proud of his accomplishments. Enough was enough. He decided he couldn't afford to lose out anymore. He came to us as Intentional Product Manager, determined to turn things around. We worked with him to refine his narrative, boost his interview skills, and truly represent the outstanding professional he is. It took work, but he got there. The result? He's achieved a prime role at a company you would have heard of and now makes over a million dollars annually. Not only did he secure the financial success he always deserved, but he also earned the respect and admiration of his peers and, most importantly, his family. But as importantly, he carries his skills and executive presence every day. --- P.S., If this resonates with you, and you: - Are a product manager - You love to take action - You are committed to career growth But if you need help in landing that ideal next role. DM me "dream role" and I will send you details. We only work with selected folks and have started to limit how many people we accept every month.

10

1 Comment

If ChatGPT was AI's "IPhone Moment", by now we're well down into the IPhone 5s moments. Lots of minor tweaks, maybe a new button or missing connector, still lots of hype for some reason, but fundamentally it feels pretty much the same. That said, don't believe the naysayers that see this as the beginning of a decline. I'm still very bullish on AI, it's just that the interesting parts are shifting from who can throw the most compute, and raw data (and legions of fundamental researchers) at the problem to who can make the advances needed to expand the technology to new uses. (Note added at writing, I just realized I'm effectively saying the opposite as what Sam Altman said this week which was "assume it will get way better" instead of find ways to live with current performance. Good. I disagree and the previous deep learning cycle backs me up). All that said, I wanted to share some interesting AI stuff that's on my radar this week that's not about new model releases (mostly): https://www.vals.ai/ Vals.AI is a startup making application specific benchmarks. Good benchmarks and better performance go hand in hand, getting out of theoretical evaluations and in to something practical is going to be the future. (I think the same thing about data: https://lnkd.in/eNepJ7Ys) https://lnkd.in/eHxbBE2e Eq-Bench is another benchmark that I hope is going to help show how silly many modern evaluations are. It achieves great predictive power for how models perform on other tests and in human evaluations by asking them to rate the emotions in dialogue samples. It's a great little test but also has nothing to do with most real applications, and sort of implies that neither do most of the other tests. https://lnkd.in/eFkAXYb7 IPEX LLM by Intel. This has been around for a while but is getting more interesting as Intel releases new GPUs and there is a push to do model inference faster and more efficiently. Check it out as an alternative to Hugging Face for LLM inference. https://lnkd.in/eFJdV_AB Pile-T5 by Eleuther AI. This work takes an old model and an old data set and looks at what happens when you train it for twice as many steps. Most interestingly, they release (if I counted right) 200 checkpoints along the way. This is mostly unheard of for big models and I'm hoping (as are they) it's going be lead to some interesting results related to interpretability, (which despite what you may have heard isn't really possible with LLMs yet). For example https://lnkd.in/eWQZCbu2 So, boring time this week for new mainstream models, lots of other cool stuff going on. Don't give up on AI yet.

14

1 Comment

Unsolicited feedback, wild-ass guesses, and dating - my three favorite subjects rolled into one! On this weeks episode of Unsolicited Feedback with Fareed Mosavat, we dive into Tinder's fresh feature, "Share My Date". 📲 Productizing Safety Steps Customers Already Take 🚀 This new tool isn't just about keeping daters safe—it cleverly integrates a common safety practice into the Tinder experience. Users can now easily share where they're heading, who they're meeting, and the meetup details with their circle, making what used to be a high-friction task just a couple of easy taps. Broadening the Tinder Universe 🌐 Could "Share My Date" be the key to unlocking a whole new user demographic? Friends of daters, matchmaking buffs, and safety-conscious singles might find themselves drawn into the Tinder orbit, not to actively date, but to participate in the dating journeys of others. It's a smart twist that could significantly expand Tinder’s appeal. Fareed, who missed out on the Tinder experience due to being married before the app’s rise, now finds a kosher reason to engage with this wildly popular app. Keeping Lovebirds in the Loop 💌 Even after users find 'the one,' "Share My Date" gives them a reason to stay on the app. This feature could transform Tinder from a dating platform to a social sharing space, helping retain users who might otherwise churn after coupling up. Wildcard Hidden Benefit - Polyamory 🎲 In my time on the apps, I’ve noticed a significant rise in the amount of people in polyamorous relationships, and thought this feature might unexpectedly benefit those in non-monogamous relationships. As societal norms evolve, this feature could not only cater to polyamorous individuals by bringing their partners into the app but might also spark curiosity among monogamous couples about opening their relationships. Spurring Viral Engagement 🌪️ Imagine the ripple effect when users start sharing their dating details. Every shared link is a mini-advertisement for Tinder, subtly inviting more people to check out what the app has to offer. It's smart, it's social, and it just might be the next big viral loop in the world of dating tech. Safety First, But Also Data Smart 🛡️ Promoting safety is a win-win, but let’s not overlook the potential data goldmine. Knowing where dates are happening and how users are interacting with this feature could drive Tinder’s strategy forward, helping them craft even more engaging and user-friendly innovations. Redefining the Role of Dating Apps 💑 This isn't just another feature—it's a strategic move that could redefine how dating apps support their users beyond the screen. By facilitating safer, real-world interactions, Tinder is possibly paving the way for dating apps to become more integral to our social lives. So, is this just another swipe or a true game-changer? It’s hard to say just yet, but one thing is clear: Tinder’s "Share My Date" has us all watching closely! 👀

80

9 Comments

In UX, I'm actually noticing the rise of the player-coach model, where you're a mix of both a manager and an IC. The job titles differ - sometimes it's a manager role but the JD notes that they expect this person to also dive into Figma to do design work, sometimes it's an IC with tactical UX responsibilities who is also expected to manage a couple of junior members of the team. This could be directly related to the many layoffs our field has faced since 2022. Companies want to get more out of each person they hire and run leaner than they might've in the past. There's also probably part of this that is related to what Dan mentions below as well: less organizational layers and middle managers overall. Have you noticed this trend in UX? 💡 What about in other fields?

2

From the scrappy pre-seed, stealth-mode team of 2 through to fast-scaling Series C/D companies with many, many millions in the bank, each startup brings its own excitement and challenges. Of course, hiring strong technical talent is *always* important, and that’s a challenge in itself. But there’s also a certain nuance around the “people fit,” especially in the earliest stages. And I’m not necessarily talking about culture add/fit. The main thing that comes to mind with this, and one of the biggest things I drill down on with candidates, is motivation to join a company at the beginning stage. It’s absolutely not for everyone, and that’s okay!! The second thing that comes to mind is being radically transparent with candidates about what’s under the hood at the startup they’re considering. There’s no point in sugar-coating or selling a dream that doesn’t exist. Even in the most innovative and exciting startup there’s bound to be some cons to joining. It’s the nature of the beast that is startup land. Give the full picture, the right candidate will be excited regardless. Getting as much as you can out in the open in your initial screens and interviews, and re-checking on how folks are feeling throughout the process, is one of the things I feel is not spoken about as often yet is so important! (+ a good way to minimize cold feet/drop outs!)

23

5 Comments

I do a lot of AI automation work, much of it involving batch processing and reprocessing of vast amounts of text. The low cost and high quality of Claude 3 Haiku (25 cents per million tokens in and $1.25 per million tokens out) has been fantastic. But today, OpenAI released GPT-4o-mini for use in API calls, driving the cost of AI automations even lower: 15 cents per million tokens in and 60 cents per million tokens out! To give you a sense of the savings: I once accidentally ran my batch processor using the high-end GPT-4o model. With 12 million tokens submitted and 2.5 million tokens returned, it cost me $97.50. With GPT-4o-mini, today that cost is only $3.30. That’s over a 96% savings! 😲 I have reviewed the Llama 3 8b + 70b Meta open-source models for these automations. The cost is lower still (the model is free, but hardware time is not). However, in my tests, the quality just isn't the same. Plus, considering the additional effort to host an API to access the Llama 3 models, my engineering time and money have so far been better spent using the low-cost models from OpenAI and Anthropic. Anyone else finding similar results in savings + quality?

35

1 Comment

Thanks for sharing our story and your POV - agree that there is never a "good" way to lose your job, but being publicly labeled as an "underperformer" has to be among the worst and most insensitive ways. Here's the link to Sarah Bregel 's story https://lnkd.in/exqXtADi

15

6 Comments

Yesterday, I told a room of Boston's top tech leaders something unexpected: I'm an awkward introvert. I shared this because every founder in that room deserved to know: even the most successful leaders sometimes struggle with networking, and that's perfectly okay. Throughout my career, I've had to debug this mindset. The hard truth? The most transformative opportunities rarely come from what you know, but from who you know. That pitch that lands, that key hire who transforms your team, that crucial piece of advice that helps you avoid a costly mistake – these emerge from networks we build before we need them. And for anyone who finds networking challenging, here's the actionable wisdom shared at our breakfast: ⭐ Partner with one trusted person at events ⭐ Prepare 3-5 meaningful questions that spark real conversation ⭐ Reach out to a few attendees before events to create familiar faces ⭐ Build simple follow-up systems that turn brief meetings into lasting connections ⭐ Have some "catch your breath" moments in your back pocket if you get overwhelmed (like a faux phone call, a bathroom break, a walk around the building) Your network isn't just about support – it's about survival and success. Every leader there had "scars on their back" and lessons that could save others months or years of struggle. Is networking sometimes transactional? Yes. Uncomfortable? Absolutely. But each interaction is an investment in your future. Your network becomes your inspiration source, your imposter syndrome fighter, your path-smoother, and often, your biggest champion. Remember: people genuinely want to help. They want to share their stories, their lessons, their connections. But they can't help if you don't show up and stretch yourself. What started as a personal challenge has become one of my most valuable professional assets. As you climb the leadership ladder, success becomes less about what you can do alone and more about what you can accomplish with others. Immense gratitude to the incredible women who shaped this discussion, especially Sam King who led the discussion, Lily Lyman our host, Thara Pillai, Kristen Craft, and the many others who shared their wisdom so generously.

105

20 Comments

Six teams. One mission. Zero department politics. Here's what happened. In under a year, we tackled what seemed impossible: completely transforming our tech stack. 5 major launches and over 25+ passionate teammates working together- we didn't just streamline systems. We redefined what cross-functional excellence looks like. 1️⃣ Your tech stack is your competitive edge. In the AI era, simplicity wins. And we are all in on HubSpot on HubSpot. The cleaner your architecture, the better your AI performs. We're seeing unprecedented servicing efficiency because every team – from Product to CS to Ops – optimized for simplicity from day one. 2️⃣ Start Together, Win Together. The foundation? 100% leadership buy-in across Product, CS, and Ops before we wrote a single line of code. We kicked off with a 2-day workshop where hierarchy took a backseat to expertise. VPs sat beside frontline managers, tech leads collaborated with business analysts. Decisions that usually took weeks happened in hours because everyone who needed to say "yes" was already aligned and in the room. 3️⃣ Trust the Experts, Move Fast Instead of getting trapped in endless planning cycles, we empowered our front-line experts to make real-time decisions. The result? 24-hour response times to critical issues and a GTM team that knew we had their back. As one team member put it: "We didn't always know how we'd solve something, but we knew we would." ✨ There are SO many to shoutout that made this program successful. Special mentions to Rebecca Armistead, Pamela Nunez, Louise Ryan, Aly Suleiman, Jordi Zacarias Llera, Cory Penquite, Maria Alejandra Franco Quintero, Jeff Love, Alex Balcerski, Amy Esposito, Anna Ishizeki, Rachel Ram Chandra, Ashley Cobane, Karolina Zielinski, PMP, Melissa King, MPH, Benjamin Bedenkop, Edward Donovan, Gabriel Bertero, Jo Crossick, Timucin Uygungil, Clodagh Joyce, Rich A., and of course David Hunt for the incredible vision and determination. And to every single person who showed up ready to win for our customers – you know who you are! What are examples of excellence in execution you've seen? Share below 👇

119

13 Comments

I couldn't recommend this more - I was reading through the other day, and the arguments resonated quite strongly with the intuitions around why a purely preferentist (or alternatively, purely utilitarian) approach to AI alignment may be flawed. This is especially important in the coming era where we are likely to see advanced AI assistants more widely deployed than ever before - and where aligning them to human values and user well-being remains of utmost importance. https://lnkd.in/dyWqkTQm

30

DEV-LESS DESIGN: This is the most important educational experiment in design and AI happening in our world 👇. The design pedagogies of Prof Donn Koh and Helen Armstrong are the current standard bearers in my opinion.

347

22 Comments

This was a fun PR! https://lnkd.in/e_SgSZT9 Occasionally, I get an urge to scratch an itch for how can I make something better for the benefit of myself and others. In this case, when looking through the NVIDIA/RAPIDS cudf docs, there is a page which lists how cudf differs from Pandas. It's a sphinx extension which is a commonly used documentation generator for libraries in the PyData ecosystem. The page lists out the differences but it's plain text. Sphinx allows you link to other docs in others project very easily. I wanted to add that here so I could quickly jump to the pandas docs to see if there was something missing in cudf and I would be fine sticking with pandas on a per project basis. This PR was built upon part-time learning over a couple of weeks and years of working in the PyData ecosystem (dask, xarray, geopandas). First I wanted to understand sphinx (https://lnkd.in/eYEWWkc8) and sphinx extensions. Next I wanted to work on the extension in a minimal and simple environment (https://lnkd.in/eyDjaAiN) where I used GitHub Code Spaces and deployed the docs on GitHub pages for quick feedback loops. I want to say a big thanks to the OSS developers who developed these tools for free (cudf, pandas, sphinx) which makes my work easier every day.

16

2 Comments

I can’t believe we used to do this to get structured outputs from LLMs. That was before DSPy, Instructor and anonLLM. It made me want to reflect on the best practices going forward. This is super important because of the rise of compound AI systems that require structured outputs to streamline communication between components. Check out my article below for more info! https://lnkd.in/ejCunhQ5

3

Whether to include LLM-based GenAI features in your data product is not a simple, binary question. Strong leadership is needed, and context really matters—particularly when we have this risk of reacting to incorrect information that looks plausible and believable because of how many LLMs tend to respond to prompts with a positive sheen much of the time. Learn more below. ⬆️ Join 2k readers: tap “Visit my website” to get my latest thinking in your inbox 🚫 Don’t like LinkedIn’s algo deciding what you see? Tap the bell under my profile banner to get all my posts ♻️ Please repost if this was useful 🎧: https://bit.ly/3W5IddY #LLMs #genAI #ai #ux #design

2

5 Comments

These AI avatars are arguably the best in the world. Which do you think is actually me: A or B? Not sure how to make a poll here on LinkedIn, but on Instagram, the ratio of correct to incorrect guesses is 2:1. I was in NYC to interview Synthesia Co-Founder & CEO Victor Riparbelli, and I had fun making this avatar. It's super easy to do on your own - just upload a few minutes of footage. —- 👋 Hi, I’m Harper, AI educator, engineer and advisor. I have two degrees in Computer Science specializing in Artificial Intelligence from Stanford University, experience teaching PhD-level AI courses there, and 4+ years building AI at Facebook / Meta. My passion is making AI useful and understandable to everyone; let me know in the comments👇 if there’s anything specific you want to learn. #aitools #aiavatar #technology #ai

139

52 Comments

I moderated a panel at Startup Boston years ago when I was learning to code, so I'm excited to return this Tuesday to speak about a topic I'm passionate about: open source. Here's a sneak peak on some of my thoughts: There's so many ways for startups to get involved in open source. Low hanging fruit is to actually contribute to existing open source projects that you use daily! So many startups turn into unicorns by relying on open source software, but open source maintainers are often undervalued and overworked. We'll explore how to support them through: - Financial sponsorship - Employee contributions - Employment-based sponsorship For startups looking to open source their own projects, I'll cover key preparatory steps: - Choosing the right license - Developing a sustainable monetization strategy - Establishing an Open Source Program Office (OSPO)  #SBW2024

42

2 Comments

There's hype and then there's this. LLMs are amazing for what they are, and can do - but increasingly folks that defend LLMs (as they are) as being able to reason sound more like parents that defend their children. We're going to need another component for reasoning that isn't what LLMs are doing today that will get incorporated in - that will be a major advancement towards intelligent systems.

12

9 Comments

Perplexity https://www.perplexity.ai/ I've noticed I'm increasingly shifting away from ChatGPT's native interface to their platform, and I'm genuinely impressed with their progress. They're rapidly rolling out new features and capabilities, and the recent Mac app launch is a particularly welcome addition to their ecosystem. Their model selection offers significant advantages, featuring Llama and Claude, though I'm eagerly anticipating potential integrations with Gemini and Grok. The current need to access different models through separate platforms creates unnecessary friction in the workflow. The Spaces feature is an absolute game-changer for prompt engineering efficiency. Unlike ChatGPT, where you need to craft prompts repeatedly, you can create template-style configurations for recurring tasks. I frequently use this for translations, having set up a Shakespeare-style English output for my Russian texts - it's a massive time-saver. Their evolution from a pure search platform to a search+LLM hybrid shows remarkable product development intuition. The seamless source linking and reference system adds substantial value, enabling deeper research when needed. While their news aggregation feature shows promise, there's clearly room for growth and refinement in this area. What's particularly fascinating is watching two distinct market approaches converge: traditional SaaS applications incorporating LLM capabilities, and pure LLM platforms expanding into conventional application features. Their focus on developing team collaboration features and enterprise capabilities demonstrates a solid understanding of market demands and scalability requirements. Looking at the search landscape, I don't subscribe to the notion that LLMs will replace traditional search. Players like Google bring tremendous assets to the table: robust data infrastructure, extensive web crawling capabilities, a trained marketing ecosystem, and deep market understanding. The future lies in integration and synergy rather than replacement. The platform's trajectory reflects a sophisticated understanding of how AI tools need to evolve - not as isolated solutions, but as integrated components of existing workflows and business processes. It's exciting to watch this transformation unfold in real-time.

11

2 Comments