Dennis Sawyers

This is going to enable stunning levels of survey fraud. AI can now literally just look at the computer screen and take the survey.

5

3 Comments

Something feels very off with Anthropic's latest model, Claude 3.5 Haiku, which you can see in the attached (data from Artificial Analysis). Anthropic says that, since the model is smarter than their best model from earlier this year, "we've increased pricing for Claude 3.5 Haiku to reflect its increase in intelligence." But if you look at comparable options, like Gemini 1.5 Flash and GPT-4o-mini, Claude 3.5 Haiku has worse quality, slower speed, and a much higher cost. This goes against long-term trends. Models have been getting smarter, faster, and cheaper in combination. Today's GPT-4o-mini is, for example, about 100X cheaper than GPT-3 (per thousand output tokens), while being faster and smarter. Anthropic seems to claim that it's worth the price because Claude 3.5 Haiku is the best of all the small models at coding. But is that really how most people will use Haiku? For coding assistance, most people will use a top model like Claude 3.5 Sonnet. Smaller models tend to be for larger scale tasks where the cost of tokens can add up. So what might really be going on? There have been rumors that Anthropic had a failed training run for Claude 3.5 Opus, which people were also expecting along with the release of 3.5 Haiku. There have also been rumors that they're behind on creating a reasoning model to compete against OpenAI's o1. And there are rumors they're raising more money. All of these pressures may have forced their hand here. It's speculation, of course, but it's hard to understand why they would do this if they weren't under such pressures.

9

The thirst of data in all organizations is insatiable. While everyone argues over which LLM is the best, pragmatic data leaders are and should be concerned about how to serve data and semantic intelligence to these thirsty models. It’s time to talk about platform interoperability. That is the surest answer to this predicament that every CDAO finds him/her/them in. It’s a topic that does not get as much of a voice as it should.

41

6 Comments

Fine tuning is probably not a smart strategy to allow LLMs to understand up to date facts, as Ethan indicates in this post with attached paper. However it's also very easy to cause "catastrophic forgetting" by overly fine-tuning on too large a dataset with too many epochs. So you don't do that. Fine tuning isn't a way to get a model to know updated facts. It's a way to get desired response formats easily (more than anything else) I personally have a pretty big problem with the use of huge context windows as a solution to hallucination. Here's an analogy... I give you the Encyclopaedia Britannica and all you want is what's the capital of Zaire. With RAG, you know Zaire is in the last volume near the front, and quickly find the answer. With "large context windows" you scan page by page from the first page, reading every word... Until you find the answer Large context windows as a solution to hallucination are environmental vandalism, as well as a waste of time waiting for a response I believe RAG still is the best solution to most "current knowledge" problems, but some types of questions still aren't easily answered using RAG. Fine tuning can help, by making models more likely to say "I don't know" (as a very simple example). There are many other approaches to improving RAG for specific types of questions and specialised domains. What do you think?

38

2 Comments

In the wake of genAI hype, developers of all stripes are striving to design and implement application architectures that make use of the advantage of #LLMs and appropriately manage their disadvantages. 2024 is year 1 for optimizing genAI applications, and today you've probably heard of Retrieval Augmented Generation (RAG) as one of the common approaches. But what is #RAG in a business context? How can it be applied to enterprise data and applications? And are there other approaches to customizing the behavior of foundation models for language like #LLMs? To get more information on the first question, check out Rowan Curran’s recently published pair of Forrester blogs on RAG and what comes next in #genAI-infused application: #RAG is all the Rage (Part 1): https://lnkd.in/eTHwdJCU The Machine is Getting More Complex (Part 2): https://lnkd.in/eWWVzDve If you'd like to learn more about the latter - adapting LLMs to your enterprise - come to my session in September at #Forrester's Technology and Innovation forum: https://lnkd.in/eWnSMNWX

6

#PET #OWNERSHIP? Reminder of proxy variables Looking at map of pet dog ownership tells… … easily mistaken story. The clear geographical delineation in graph (matching the cold war Iron curtain line) can easily sway you into cultural root cause. 🤦🏻‍♂️ And that would be a great example of the correlation fellacy. Because 25 years after the Iron curtain is down is 2-3 generations of the dogs, long enough time for the political factors to wear off, if they ever had effect. 📖 The real learning of this graph is that there might be hidden factor that is not obvious and still it has profound effect on the actual “target variable”. 🐕 In our dog situation the underlying hidden root cause for the pet ownership is the share of agriculture on total labor market 1.5-2 generations back. Because if you had a farm, you were more likely to get used to living with animals or even needed some (shepherd dogs anybody?). The 2 generations lag is important, because you don’t make pet purchase decisions until you live in your own household and that is roughly, when you start your family (hence when you pass on the generation baton). 🔗 This phenomena is also known as #proxy #parameter and has significant impact also (in positive means) in #MachineLearning. Because proxy parameters are strong predictors, but are insights that markets tend to oversee. So if you know them, proxy parameters give you edge. 💡 Now you know the phenomenon I have a #puzzle for you: It is proven by data in many insurance companies (I worked on one of such studies) that #Zodiac sign is very strong predictor of the #car #accident rate. Yes, really true! Can you guess what it is proxy parameter of?

3

My first conversation with new hires: We're excited to have you. There were other good candidates but you were the best. Some day, however, you are going to leave. If you find a better opportunity somewhere else, there's no hard feelings from me. Hell, recruit me to there. But while we take a paycheck, we owe 100% of our effort to making this the best place it can be for each of us. I'm going to be pissed if you leave over something I could have fixed. Don't check out. I can't promise I'll always be able to get you the work and experiences you are looking for, but I'll do whatever I can. Talk to me. If you have issues, I need to know. If your issues are with me, bring them to me anyway. Or talk to my boss, or the head of HR, or the CEO. You will know what you are supposed to be working on and have clear line of sight from that work to the benefit for the business and our customers. Hold me accountable for that. If you lose sight, let me know and we'll figure out if you just need some more information or if we need to change direction. You always need to be able to answer these three questions: 1) What are you working on? 2) Why are you working on it? 3) How will you know when you are done? What do you need from me?

634

35 Comments

Here's why OpenAI o1 is a big deal. Up until now, the only way to make a model smarter was through training. You had to train on more data with more compute. These training runs are expensive. ChatGPT 4 cost about $100 million to train. Future models are expected to cost $1bn to train, then $10bn. You get where this is going. Once the model is trained, it's pretty much as smart as it's going to get. What they've done with OpenAI o1 is develop a technique where it can get smarter at run time (inference) just by pondering the problem longer. How much smarter it gets scales with how long it thinks. This is a breakthrough.

21

8 Comments

To me, RAG is a knowledge management & search problem. vector search is one strategy, but should not be the only strategy as different question calls for different search strategy https://lnkd.in/gkBYeQJN

19

New research shows that, indeed, o1 models are better at planning than even the best LLMs—but we're just getting started. o1 models crush LLMs on short planning tasks with low complexity, but still struggle with longer, more complex, and more unclear tasks—and with knowing when a task is unsolvable. Still, this seems like significant progress. We also have a new term to consider adding to our AI lexicon, "large reasoning models" or "LRMs." Not sure if that's been widely used already. Paper is here: https://lnkd.in/dzgsdkt7 Credit to people on X for circulating this, as it's making the rounds there.

2

There has been a lot of hype around a semantic layer or metrics layer as an independent and separate part of the analytics stack. We've been hearing about this for years, but I think nobody in the data ecosystem has really delivered on the promise of an independent semantic layer outside of our friends at Cube. I'm excited that we're launching our Hashboard partnership with Cube today. Now you can easily explore and find insights in the metrics you define in Cube. We'll be hosting a webinar tomorrow to show off how the products work together - link below Hashboard and Cube have a strong shared vision for governed and consistent metrics. While Hashboard has an integrated semantic layer (Hashquery), the purpose is just to drive consistent metrics and exploration inside of Hashboard and to make the most compelling reporting system we can. There is still a missing link for plugging metrics into other tools in the ecosystem. I see two basic needs for a *separate semantic layer: 1️⃣ SQL is not particularly composable, it's hard to share calculations (I've written a lot about this) 2️⃣ The analytics stack has a bunch of missing links: tools not being able to interoperate. Power BI, probably one of the most popular BI tools doesn't work particularly well with Snowflake for example. Hashboard is certainly not going to help you connect Power BI and Snowflake anytime soon. There are a few requirements for a semantic layer to be useful as a separate part of the data stack (these problems are *hard): 1️⃣ It's ultimately a network effects business: you need lots of integrations both upstream and downstream for all of this to work. 2️⃣ It's hard to get visualization tools to want to adopt you until you have lots of upstream integrations and customers. So you have to make integration really easy. I think Cube's secret sauce is its sql api: you can just write sql against Cube as a visualization tool which means that you don't have to do a highly customized integration. Their semantic layer speaks sql which is pretty familiar for all of the downstream tools in the ecosystem It has been delightful to get to know and work with the whole cube team Brian Bickell Artyom Keydunov and Tony Kau!

50

4 Comments

Most of the internet is generic BS. Gen AI is trained on the internet. Therefore, Gen AI produces an awful lot of generic BS. I'm going to submit this as an SAT question. Don't get me wrong - Gen AI is an amazing tool, but don't mistake using it for learning or for being original. My class finished this week, and I'm grading the last few assignments submitted late. There's a lot of Gen AI. You know how I can tell? 1. Lack of focus. A one paragraph answer is structured like (and as long as) an essay. 2. Generic answers. Technical details generic are unrelated to the problem at hand. 3. Lack of citations. Always know where your information comes from. I'm not saying anyone should cite sources in social media posts or business presentations, but the first two are very relevant. Gen AI is a tool to enhance creativity and learning, not a replacement for it.

5

1 Comment

I'm happy to announce my next role: I'm going to open a boot camp to teach other people how to open data boot camps! I noticed that the TAM for this is bigger than for data boot camps. A new HBR survey shows that there are now more people running data boot camps than there are people with jobs in data. Sign up and you'll learn: 1) How to get past your imposter syndrome over getting paid $500k to do a job that didn't really add value so you quit after 1.5 years. 2) The importance of "ex-_____" in your LinkedIn headline. 3) Making sure your students don't find all those high quality, free YouTube videos by brilliant Indian dudes. 3a) Emphasizing that those guys are only "ex-FAANG offshore contractors" so totally different. 4) The art of the self-congratulatory LinkedIn post where you point out that making 7 figures as your own boss is so much better than when you were making mid 6 figures working for FAANG. It's the tech bro version of selling stock tips while standing in front of a Ferrari. Classy. My next venture is also coming up. It will be in the boot camp for Web3 boot camps space!

137

21 Comments

I often find myself wanting to reference a "greedy algorithm" but know I will be misunderstood by the anti-market commentariat that makes up a surprisingly large part of LinkedIn. Greedy algorithms are those that select locally optimal paths without regard to the big picture. They are useful because they are low complexity and require very little data. The classic example is making change. Subtraction is hard, so it isn't too critical to say that most cashiers couldn't make change if they had to use it. Instead, you make change by adding as many quarters as you can, then dimes, nickels, and finally pennies. 82 cents takes 3 quarters, 1 nickel, and 2 pennies. In general, finding the smallest number of coins to make change would be equivalent to the knapsack problem and therefore is NP-Complete, so the greedy algorithm is a big win. Not all greedy algorithms guarantee global optimization. In fact, lets look at the above 82 cent example when we run out of nickels. Using our original algorithm, we'd grab 3 quarters and 7 pennies, for 10 total coins. But we can make change with 2 quarters, 3 dimes, and 2 pennies, totaling 7 coins. The coin denominations are selected to make the greedy algorithm find the global optimization. I'm amazed at how many of life's problems are caused by the mismatch between the locally optimal solution and the global one. I recently posted about Boeing (and most other companies) eschewing vertical integration to focus on only the places they add unique value. Every step down this path makes sense, but some day you wake up and find out you don't actually do anything anymore, because you've shed the whole business one step at a time. This post is actually about investment advice, though. Or more specifically how index funds are going to crash the stock market. Index funds are the best choice for each individual investor. Low fees and high diversity are shown to win time and again. But most money is being passively invested this way. And that means when NVIDIA goes up, we are all holding more of it to reflect the larger share of total market capitalization. Prices are supposed to be market signals, but the end state of indexed investing is that stock market price only indicates stock market price. What should you do about this? Well, the best move for you is to be in index funds, because the alternative is sub optimal. But the global outcome after the aggregation of all these locally-optimal decisions is sub optimal. Very suboptimal. We're out of nickels and it isn't going to end any better for the stock market than it did for Boeing.

35

14 Comments

I wonder what the click rate is for those LinkedIn AI suggestions on the bottom of posts. Corollary: I wonder what percentage of those clicks are accidental and whether it is three digits.

3

5 Comments

Interesting article. With the ever increasing size of context windows, massive prompts could potentially outperform fine-tuning for LLMs, based on a study from CMU. That means depending on situations, one can choose to front load their cost to fine-tune LLMs, or to have larger prompts and pay more per transaction to get to similar performance. https://lnkd.in/euFTP8QS

5

I'm looking for better options to NER on .html files. I'm currently using BeautifulSoup (lxml parser)-> stanza (which seems to perform better than Spacy), but I'm getting a lot of junk classified as an entity. I'm trying desperately to get better off the shelf results with an understanding that there will be a post extraction filtering phase to further refine. thanks! #nlp ##datascience #ai

9

10 Comments

Whoever claims building RAG systems is easy has never built one at scale. 🤯 There is a significant gap in the literature and existing frameworks. RAG architectures are designed for POCs, not for hundreds of thousands of documents or supporting tens of thousands of users. Designing these systems requires a cross-functional team with software engineering and data science skills blended with a product mindset. Earlier this summer, I spoke at the Databricks AI Summit, highlighting guiding principles we are taking at CVS Health for building enterprise-grade RAG systems. Every company is unique, and while there is no one-size-fits-all solution, here are three principles that will help you on your journey: 👉 Spend time on discovery—It's tempting to dive in and start building immediately, but you may get lost in the desert without doing your due diligence and understanding what delivers the most ROI. Our team spent approximately a quarter on discovery. 👉 Design for modularity—The LLM space is changing rapidly, and you want to build reusable components that can easily be swapped out or upgraded when something better comes along. 👉 Humans + Machines—Technology alone can not solve the knowledge search problem. Develop processes and governance to help teams curate the data, which will be ingested into your system. These lessons are just the tip of the iceberg. Watch my full Databricks presentation and leave your comments and questions below to learn more tips and tricks. #RAGSystems #EnterpriseAI #DataEngineering #MachineLearning #AIArchitecture #AI #DataScience #LLM #SoftwareEngineering #AIAtScale #KnowledgeSearch #TechLeadership #AIFrameworks #AIInnovation #DatabricksAISummit #ProductMindset #CVSHealth

62

5 Comments

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2