Saharsh Agrawal

Early-stage fundraising is getting complicated due to SAFEs. Should they be limited to early rounds? • Angel Rounds (SAFEs): $150K-$200K raised, small valuation caps, light dilution. • Small Pre-Seed (SAFEs): $680K raised, $9M cap, 6%-8% dilution. • Big Pre-Seed (SAFEs): $1.5M raised, $12M cap, 11%-13% dilution. • Seed (Priced vs SAFEs): ~$3.3M raised, $13.7M valuation (priced) vs $18M cap (SAFEs), slightly lower dilution with SAFEs. • Post-Seed Extensions: More common, but costly in equity. #Fundraising #Startups #VC #Investment

7

The pricing model for most LLMs feels very misaligned on the surface. You pay per token, and the LLMs does not really have a great way to generate just the right amount of tokens. Here's an example: I ask an LLM to generate and run some code for me. The model will usually fail to: 1) open the file in the right format; 2) import the right libraries; 3) cast types in the right way. All these errors consume precious tokens and pollute the context window. Because of this, some of our pilot partners spent up to $2 per execution just to see the LLM fail. I'm sure LLMs will improve over time, but in the medium term, this is a real challenge that impairs adoption of AI in mid term. You know me, and I'm a tease: we were able to bring our partner's costs down by about 83%!

10

1 Comment

Yesterday's code is tomorrow's prompt Meta recently proved it with 607 words Most think prompts are quick commands But that world is dead Look at Meta's AI system prompt: • 607 words of detailed specs • Technical requirements • Usage parameters • Security protocols • Edge case handling This isn't just a prompt It's a full product spec 2021: • "Hey AI, write me code" • Basic Google-style queries • Surface-level results 2024: • 600+ word engineering docs • Complex system architectures • Proprietary prompt chains The shift? Massive Your company's next 10x engineer won't master Python They'll master prompt engineering Why? Prompts are becoming: • Core IP assets • Protected trade secrets • Competitive moats • Strategic weapons • Engineering blueprints In last 2 years: Product specs → Code → Prompts Think about it: The most valuable IP at tech companies isn't their codebase It's the prompts that generate it

41

7 Comments

Running llama 3 by meta does not require you to have sophisticated system setup. I have set it up on my personal laptop which happens to have a GPU with it. I have used llama.cpp to run the int4 and int8 quantized model of llama 3. Running a language model for your personal use does not require you to have cloud access such as azure. I have demonstrated one use case of deploying personal language model. During my research purposes It so happened that chatgpt was very hard to access either cause it was down or the site simply would be buggy during which I used the llama model deployed by https://lnkd.in/gWfkaKZA where I think such systems can be deployed by organizations for internal research purposes. Below is the link to the steps I followed in deploying and comparing the int4 and int8 quantized model. https://lnkd.in/gzShCPCD

21

5 Comments

[Just Dropped - the "200b Parameter Cruncher Macbook Pro"] - New #Macbook Pros just dropped by Apple with the claim that it can "easily interact with LLMs which have 200 billion parameters" with the 128GB vram of the Macbook pro. What does that mean? With the right configuration, you'd be able to load the most demanding #LLM's locally and interact with them in all sorts of interesting ways. Keep in mind that 'bigger isn't always better' when it comes to the parameters of the models themselves, today's 8B models can often out perform 100B models from just 6 months ago, however when it comes to getting the most out of context windows AND medium size models this sort of horse power is a game changer... basically with a reasonably powerful model (like #Minstral 8B) you could likely take full advantage of it's 128k token context window - so, you could include a few hundred pages of text IN the prompt and you could interact with it all in context at a reasonable interaction rate (T/S). In english - Local LLM's are going to get a lot more usable, developers who are making bleeding edge software are going to be able to leverage that in very interesting ways! I'd love to have a local knowledge base management software which used Minstral 8B + Minstral Embed locally to fully embed my book collection, my article collections, all my notes, and allow me to interact with it FAST... it can be done now, but this new #M4 Pro processor is perfect for the task, and you can put it in your backpack. What a time to be alive. Simply claiming to be able to run 200b LLMs locally "easily" is a huge claim to anyone who has tried on a computer that doesn't have a beefy GPU. Some nerdy specs to consider... Average speed (tokens/s) of generating 1024 tokens by GPUs on LLaMA 3. - M1 Max: 34.49 tokens/s (8B Q4_K_M model) - M2 Ultra: 76.28 tokens/s - M3 Max: 50.74 tokens/s - M4 Max - Projected (tablecloth math) -> Text Generation: ~96-100 tokens/s (8B Q4_K_M model) -> Prompt Processing: ~1,200-1,300 tokens/s This projection is based on: - The 1.9x GPU performance improvement over M1 Max - The increased memory bandwidth (546GB/s) - The 3x faster Neural Engine - Historical scaling patterns from M1→M2→M3 GPU landscape: Projected speed: ~96.41 tokens/s Would place M4 at about: 67% of H100 PCIe performance (144.49 tokens/s) 74% of RTX 4090 performance (127.74 tokens/s) 26% faster than M2 Ultra (76.28 tokens/s) 90% improvement over M1 Max (34.49 tokens/s) Would represent: ~17% of H100 PCIe performance (7,760 tokens/s) ~19% of RTX 4090 performance (6,898 tokens/s) ~26% improvement over M2 Ultra (1,023 tokens/s) ~262% improvement over M1 Max (355 tokens/s) Now, the real question is, with thunderbolt 5 + NVIDIA's 5090 series coming along, what kind of madness could we see in external GPU's if software support comes along? #ollama

6

2 Comments

Task-specific models and routing architectures are the future. Robert Nishihara, co-founder and CEO of Anyscale, is seeing hybrid strategies emerge, where businesses are evaluating different types of models for different use cases. With the rapid advancement of AI tech, it won't be long before open-source models are "good enough" for the majority of business tasks. The natural progression from here is to develop smaller, task-focused models using proprietary company data. As Robert puts it: "You're going to need small, fast models. And task-specific models have a huge advantage when it comes to cost and speed because you can achieve the same quality with a smaller model if it's specialized." Another interesting development: the growing use of routing architectures. Imagine a classifier as a kind of traffic controller for your AI queries. It assesses the complexity of each query and directs it to the most suitable model: simpler requests are handled by smaller models, while complex ones are passed on to larger, more sophisticated models. “If you do it right,” Robert says, “you can achieve something like the cost of the smaller model with the quality of the more advanced model.” Dive deeper into AI architecture trends in my full discussion with Robert on the B2BaCEO podcast here: https://lnkd.in/ghigvERn

46

3 Comments

I haven't gotten a lot of hate in the past 5 years, but this thread was almost exclusively negative. Link: https://lnkd.in/eypKrGrR This Reddit post was asking if Analyst Builder was worth it - this was back when it was first launched its Beta Phase. I think some of the criticism in the post is actually fair when we first started out. We only had a few questions, we had bugs and issues, and only courses on SQL and Python. When we first launched I know other platforms were better, but after 9 months we are passing our competitors. Since then we have grown a lot, adding more features than a lot of other platforms, better tailored questions for Data Analysts, and courses on SQL, Python, Pandas, Excel, Tableau, AWS, and Azure. Honestly, I think they're better than all of the courses I took when I was first starting out. When you're building something in public, especially when you're competing against companies that have been around for 10+ years, there's going to be a lot of comparison. Others focused on me specifically - I actually think that feedback is more fair. I started my channel when I only had about 3 years of experience as a Data Analyst. I was learning and teaching people online as I learned myself! I know some people want someone who has 25+ years teaching them, but I think I did fairly well. As I grew as an Analyst my content evolved and changed with me, but my heart was always to help people. All that being said, it's humbling to hear negative things about yourself or something you're building. You can't please everyone, but hopefully there are more people who do benefit from what I'm putting out.

714

144 Comments

As LLM's get increasingly more sophisticated, with OpenAI's o1 model now including reflection before answering, I am increasingly investigating the emergence of autonomous AI Agents. My path here was somewhat circuitous in that I was going through Y Combinator's Summer 24 cohort which had a plethora of AI use cases that were very industry specific. I found that curious in that earlier SaaS offerings should have already automated and/or digitized most of these process, and I couldn't figure out why AI startups would focus on basic workflows such as inventory management and book keeping. As I dove into the matter, with the help of my now indispensable Perplexity, I landed on a very insightful post by Bessemer Venture Partners (https://lnkd.in/gHtQQRha). The thesis is that Vertical AI companies are going to be magnitudes bigger than general solutions, especially as they focus on industries that were earlier inaccessible to SaaS given very high language based workflows. Within that thesis, there was also a mention of AI Agents and how these would come to dominate workflow automation going forward. This neatly tied in with a great conversation between Peter H. Diamandis and Emad Mostaque (https://lnkd.in/g-ccxbma) that talked about 10 billion agents!! I have spent the last few weeks diving into AI Agents with the help of content from interesting voices such as Adam Silverman (Hiring) and David Ondrej. Over the next few months, I will try to create and deploy AI Agents on some basic use cases to get hands on experience on these systems. In the meantime, I will close this post with a link to a video from Andrew Ng, a pioneer in AI systems that provides a short overview of AI Agents (https://lnkd.in/g8v4pUQu). I hope you enjoy the content and wish everyone a productive week ahead.

8

WELCOME TO THE "AI RAT RACE" Great piece from CNBC reportedly giving the views of a number of software engineers working on generative AI projects for Big Tech and startup firms. The overriding impression given here is one of CEOs and VCs throwing vast sums of money at LLMs in a blind panic at falling behind (what? they don't know!), in the hopes that someday (before next earnings call!) they build something, anything, useful. Or at least that they can slap an AI label on. All without having much idea what problems they're trying to solve or what competitive advantage they may have. Or, fancy the thought, how they're going to make a competitive rate of return on their investment. A handful will surely succeed... in something... someday. But of the hundreds of billions already or soon to be invested in gen AI technology, my guess is much of that is likely to be wasted. And to accomplish this great feat of capital incineration (so soon after the last exciting episode!), we are burning actual COAL (delaying the retirement of coal plants) in order to generate enough electricity to power all those new data centers. Why does this remind me of crypto... [CNBC] An AI engineer at Microsoft said the company is engaged in an “AI rat race.” When it comes to ethics and safeguards, he said, Microsoft has cut corners in favor of speed, leading to rushed rollouts without sufficient concerns about what could follow. The engineer said there’s a recognition that because all of the large tech companies have access to most of the same data, there’s no real moat in AI. ... Regardless of the employer, AI workers said much of their jobs involve working on AI for the sake of AI, rather than to solve a business problem or to serve customers directly. “A lot of times, it’s being asked to provide a solution to a problem that doesn’t exist with a tool that you don’t want to use,” independent software engineer Kolman told CNBC. The Microsoft AI engineer said a lot of tasks are about “trying to create AI hype” with no practical use. He recalled instances when a software engineer on his team would come up with an algorithm to solve a particular problem that didn’t involve generative AI. That solution would be pushed aside in favor of one that used a large language model, even if it were less efficient, more expensive and slower, the person said. He described the irony of using an “inferior solution” just because it involved an AI model. A software engineer at a major internet company described much of the work in generative AI today as an “extreme amount of vaporware and hype.” Every two weeks, the engineer said, there’s some sort of big pivot, but ultimately there’s the sense that everyone is building the same thing. https://lnkd.in/eEqQ4rEJ

1

Prompting methods that worked for models like GPT-4 might not work for reasoning models like o1. The authors of MedPrompt—a prompt engineering framework that leverages multiple methods such as few-shot prompting, chain-of-thought, and ensembling—have re-run their experiments evaluating o1-preview, compared to their initial study which used GPT-4. We wrote about Medprompt when it first came out almost a year ago https://lnkd.in/eyT9XBPg The new version of the study shows how prompt engineering methods like few-shot prompting perform on o1-preview versus GPT-4. This data is critical as we begin to establish prompt engineering best practices for these new reasoning models. Results are below, showing that 5-shot prompting resulted in a significant decrease in performance. This aligns with OpenAI official prompting guide for reasoning models, which advises including only the most relevant information to prevent the model from overcomplicating responses. TL;DR: Few-shot prompting might degrade performance on reasoning models, but you should test it based on your use case.

23

3 Comments

Managing production software, and especially "root causing" issues, whether at the infra or app layer, continues to be a growing challenge despite the success of companies like Datadog. I have looked at applying deep learning to this problem since 2017 when we worked with Gurashish Brar & Mohit Gupta to incubate Opas.ai. The team was exceptional, and they even hired a great SDR (me 😜 ). We learnt the hard way that it was too early and #AI was just not ready. A lot has changed since in terms of data infrastructure (check out Conviva's Operational Data Platform), causal inference models and LLMs. Jaya Gupta and I are excited to take another whack at this market, which we call #AgentSRE to distinguish from the outdated #AIOps framing. If you are building an #AgentSRE company, ping us. https://lnkd.in/gvBgfN7k

45

6 Comments

In 2006, the Microsoft Zune was unable to gain traction versus Apple’s iPod due to poor marketing and lack of developer support. It’s difficult to disrupt competitors who are well-entrenched since their product is sticky, customers have a strong passion for the product, and/or the company has great distribution. I explain more about lessons from product flops in my blog: https://lnkd.in/gBAuMS8m My hobby of collecting artifacts from failed companies and products can be seen at www.failure.museum. I feature over 1000 items and their story of failure. Please check out the website and let me know if I'm missing any failed companies or products. Failure Museum

27

7 Comments

Excited to share that I'll be giving two presentations at KubeCon & CloudNativeCon North America in November in Salt Lake City! 1️⃣ Unlocking Potential of Large Models in Production - with Adam Tetelman https://sched.co/1i7ns 2️⃣ WG Serving: Accelerating AI/ML Inference Workloads on Kubernetes - with Carlos Eduardo Arango Gutierrez https://sched.co/1hoy0 There are many other great talks on KServe and Kubeflow from my wonderful colleagues and collaborators. Check them out! - Engaging the KServe Community, the Impact of Integrating a Solutions with Standardized CNCF Projects - Adam Tetelman, NVIDIA; Taneem Ibrahim, Red Hat; Johnu George, Nutanix; Tessa Pham, Bloomberg; Andreea Munteanu, Canonical https://sched.co/1i7r4 - Optimizing Load Balancing and Autoscaling for Large Language Model (LLM) Inference on Kubernetes - David Whyte-Gray, Red Hat https://sched.co/1i7lb - Democratizing AI Model Training on Kubernetes with Kubeflow TrainJob and JobSet - Andrey Velichkevich, Apple & Yuki Iwai, CyberAgent https://sched.co/1i7nV - Elevating Kubeflow Spark Operator's Future: Best Practices and Enhancements - Vara Bonthu, AWS & Chaoran Yu, Apple https://sched.co/1i7oq - Best Practices for Deploying LLM Inference, RAG and Fine Tuning Pipelines on K8s - Meenakshi Kaushik & Shiva Krishna Merla, NVIDIA https://sched.co/1i7rc Link to the full schedule is here: https://lnkd.in/gyDQAC9c Cloud Native Computing Foundation (CNCF) #KubeCon #K8s #CloudNative #AI #ML #ModelServing #Kubernetes

124

1 Comment

🔥 Big news in AI! Google just unveiled Gemini 1.5, and it’s going to shake things up. While most models max out at around 128,000 tokens, this beast can handle close to 2 million tokens! That’s like processing 16 books, 100,000 lines of code, or 10 years of messages in one go. What’s even cooler? Google is inviting creative minds to propose groundbreaking ways to leverage this model and offering $100,000 in prizes for the best ideas, with a top prize of $25,000. 💡 Whether you’re into AI or just curious about its potential, this is an opportunity to explore and make an impact. Check out the details here 👉 https://lnkd.in/dprgwGT5 Good luck to anyone taking part! #AI #GoogleGemini #Innovation #AIChallenge #Creativity #MachineLearning

5

The people that make the boldest predictions about technology are the ones that understand it the least. There's a common prediction among business people that LLMs will free them from a heavy reliance on technologists because LLMs can write all the code needed. This is just like thinking that doctors are obsolete because nurses can administer the shots. Technology is a multi-layered mechanism that translates hypothetical solutions into practical ones. What many people may not realize is that, within all the layers, writing the code is at the bottom of the food chain. Having a vision doesn’t mean an LLM can jump directly to building and operating a fully functional Product from that vision alone. Your Ideal Customer Profile, problem to solve, and unique differentiation shape your hypothetical solution. Add a go-to-market plan, and you have a lightweight Business Strategy. You can now derive a Product and a Technology Strategy. The role of the technologist is to define the Technology Strategy and establish mechanisms for seamless execution, all with the focused goal of bringing the Product to life. Technologists have failed for more than two decades to educate on what their role is in the business and the value they bring, oversimplifying it and underselling it. We're all guilty of this! I don't blame the business people excited about these predictions. It's human nature to want to be self-reliant and move fast. But I think they will get disappointed. Technologists need to improve their ability to communicate their value, while business people should deepen their understanding of Product beyond sales jargon and KPIs. And they shall live in peace. #LLM #AI #Technology #Business

32

4 Comments

Great podcast on: More than 1/4 of Google's Code Now Generated by AI. More than a quarter of Google’s new code is now generated by AI, signaling a major shift in software development. As GitHub Copilot adds models from Anthropic and Google, developers are gaining more options than ever. This trend suggests that AI is set to redefine coding, making software creation faster and more accessible across the tech industry. https://lnkd.in/eNxvbJr9

35

NVIDIA's Jensen said "ChatGPT democratized computing, Llama2 democratized generative AI" ...and now Llama3 is the next level 🔥 It's fascinating to watch Meta's strategic moves. With Llama3, they prioritized training on a massive 15T token dataset to pack all into a lean 70B param model, instead of building a massive model. This allows Llama 3 to match trillion+ parameter models like GPT-4, but at 1/10th the compute, storage and inference costs! 💰 This technique was published by Google DeepMind a few years ago https://lnkd.in/gx7VU8aA Meta is not a dark horse in AI anymore. They might be the top dog. https://lnkd.in/gRRJYuxt #llama2 #llama3 #llm #chatgpt #gpt4 #ai #ml

16

5 Comments

Whatever this shit, engineers need to show some backbone and refuse to commute 40 miles in bumper to bumper traffic congestion to build it for him. Particularly applies to engineers in Bay Area. Computing paradigms are meaningless when the need is for a new transportation paradigm that addresses problems like congestion and safety. I for one is quite nauseated with this over innovation in computing when the human needs are elsewhere. How does that matter to me if Andrej Karpathy’e LLMOS deals with tokens or bytes - it’s not my interest to aid him in his mental masturbation. We need a new operating system for the roads (particularly highways) to improve operational efficiency, relieve congestion, improve safety and user experience. Operating System needs to get back to its fundamentals. How about moving people instead of bytes or tokens with efficiency and basic dignity if not user experience? The context window is a platoon of “real” heterogenous vehicles including buses and cars moving at highspeed insync on a highway.

2

1 Comment