Kieran Taylor

We’ve run almost a billion inference requests at Beam Here are a few things we’ve learned about real-time inference APIs 1. For small models, you can often run multiple workers in parallel on the same GPU and increase throughput 2. Weights should be cached on disk or in distributed storage. You don't want to be downloading large files from cloud storage during an inference request 3. Cross-region latency matters. Regional caching is hard and requires a lot of planning 4. You can't use the same autoscaling strategy for every model. You have to see how your workload performs in the real world and choose a strategy that makes sense When you run APIs powered by large ML models, performance isn’t guaranteed. What other strategies have you employed in your ML pipelines to optimize performance?

37

1 Comment

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

It looks like the cost of model training is going into the stratosphere! Zuck said Llama 4 will take 10x the compute of Llama 3.1. I estimate Llama 3.1 cost about $60-70M to train. Meta reported it took 16,000 GPUs that you can rent at about $2,500 per GPU per month, or if you buy and manage the chips/servers yourself at scale it costs about half that. So $1,250 per GPU x 16,000 GPUs x 3 months of training is $60M. Llama 4 will take 10x the compute, so we are looking at about $600M. That is some serious cash.

22

7 ways to slash latency in System Design: When I worked on Meta's distributed data store, managing massive data wasn't the most challenging part – fighting latency was. After years of trial and error, I'm sharing 7 proven best practices for building low-latency systems. 1. Choose the correct architecture Swap slower monolithic architectures for modular, scalable systems that cut latency by operating independently. 2. Optimize data management Choose SQL for structured data or NoSQL for flexibility, and use indexing, sharding, and replication to speed up data retrieval. 3. Improve network design Reduce data travel with CDNs, load balancers, and edge caching. 4. Use efficient communication protocols Use fast protocols like HTTP/2, WebSockets, and gRPC to streamline communication and cut round-trip times. 5. Optimize your code Use efficient algorithms, data structures, parallel processing, and I/O reduction to speed up execution and reduce bottlenecks. 6. Use high-performance hardware Opt for SSDs over HDDs and low-latency cloud services like AWS Global Accelerator for faster performance. 7. Implement smart caching strategies Cache at multiple layers with LRU/LFU eviction and sync updates to avoid stale data. Read the full blog for a closer look at these 7 practices and how to implement them. (Your system—and users—will thank you). https://educat.tv/3B5Dh0N #SystemDesign #SoftwareEngineer

20

Carta's data is ....off? Coastal Recruiting communicates with hundreds of engineers weekly, primarily in New York and San Francisco. We consistently encounter software engineers with 2-6 years of experience requesting base salaries between $180,000 and $250,000, plus equity. Perhaps "misleading" is a better term, as I frequently hear founders saying, "But Carta indicates we can hire this engineer for $140,000 to $160,000." We then observe Seed/Series A companies paying $200,000 to $300,000 for top talent. The disconnect may stem from the fact that companies raising funds from Sequoia, Khosla, Founders Fund, Lightspeed, a16z, KP, Greylock, Index, NEA, Lux, and others are not seeking what 95% of the companies providing data to Carta are seeking. These companies are looking for the top 0.01% of engineers. I say all this while also acknowledging my appreciation for Carta. I value the data and charts it provides, but it's important to remember: "There are three kinds of lies: lies, damned lies, and statistics."

26

9 Comments

Early-stage fundraising is getting complicated due to SAFEs. Should they be limited to early rounds? • Angel Rounds (SAFEs): $150K-$200K raised, small valuation caps, light dilution. • Small Pre-Seed (SAFEs): $680K raised, $9M cap, 6%-8% dilution. • Big Pre-Seed (SAFEs): $1.5M raised, $12M cap, 11%-13% dilution. • Seed (Priced vs SAFEs): ~$3.3M raised, $13.7M valuation (priced) vs $18M cap (SAFEs), slightly lower dilution with SAFEs. • Post-Seed Extensions: More common, but costly in equity. #Fundraising #Startups #VC #Investment

7

I didn't sleep last night. Because Stability AI's Stable Diffusion 3.5 just dropped. And it's blowing my mind (as usual). An 8 billion parameter model? Yes, please. I've been playing with the Large Turbo variant. high-quality images in just 4 steps. This isn't just an upgrade. I believe It's a leap. What's new? → Query-Key Normalization for better training stability. → Enhanced customizability for fine-tuning. → Superior prompt adherence. But wait, there's more. The Medium model supports up to 2 megapixel resolution. On consumer hardware. Yep, you read that right. AI-generated art is more accessible than ever. And the licensing? Free for non-commercial use. Free for commercial use if you're under $1M revenue. Stability AI is making sure we all get to play. Sure, there are trade-offs. More variation with different seeds. Non-specific prompts might surprise you. But that's part of the adventure. We get to explore. We get to create. And we own what we make. As a bootstrapping founder, this is gold. Tools like these level the playing field for all of us. For us college dropouts. For the dreamers. For anyone willing to dive in. I'm excited. Scratch that. I'm f**king thrilled. Time to see what we can build with this. Who's with me?

12

1 Comment

Stand on the platform or jump onto the fast-moving train? Jensen Huang, founder and CEO of Nvidia, advocates for immediate action to keep up with rapidly growing trends. Take a page from Huang's playbook and don't just observe, act. Be the engine, not just a passenger. Intrigued? Learn more here: https://lnkd.in/eRKZbpvp What are you doing to drive change and #transformation ?? #databricks #nvidia #genai #generativeai

5

Excited for the SF DataFusion meetup next Wednesday! I'll be giving a talk about how Arroyo implements dynamically-loaded UDFs. Because Rust lacks a stable ABI, this is harder than it sounds—different compiler versions or even changes to flags can break code loading. But we don't want to recompile our entire engine just to use a UDF. This gets even harder if we're trying to use async across a UDF boundary (which Arroyo has to support to enable things like HTTP calls, database lookups, and model inference in UDFs). How do we do it? You'll have to come to the meetup to find out. But I'll give you a hint: it involves C 😱 See you there!

34

5 Comments

Last year, Huggingface deprecated their InferenceApi client and switched inference calls to InferenceClient. Simple enough. But see, Langchain's HuggingfaceHub package still called InferenceApi. So anyone running langchain.llms.huggingface_hub received an future warning error that InferenceApi was depracated and to use InferenceClient instead. The problem was that unless you went searching around into the details of the code langchain.llms.huggingface_hub package, you had NO idea what this error meant and whether it was an important warning to heed. So one would think Langchain would resolve this quickly, right? No. This was not resolved universally until a January 2024 update. Until then, people were using patches to get around this error. AND if you're thinking, "Wow! Well, glad that's no longer a concern" think again because out of curiosity, I asked ChatGPT 4o list imports for me to create multi-agent AI using langchain and inferencing the mistral API and wouldn't you know it, it suggested I use the deprecated InferenceApi client. Now, can you use the InferenceApi client and it still work? Sure. However, that future warning error gives lots of engineers who have not seen that error before pause AND it would likely be an issue when considering if one's code is production ready. When one AI company changes a client name and another AI company is using their own client to call on the original AI's client, this creates another dependency. One that a LOT of engineers building AI are not aware of unless they get used to seeing these errors over and over and over again. Little things like this are keeping great AI model demos from becoming great AI products.

45

6 Comments

Agreed! That's why we've been expanding our 3D asset management capabilites at VNTANA. We completely agree devs should stop wasting time which is why we have: -Intelligent Optimization to automatically reformat and resize 3D assets -Robust search and meta data tied to assets -Approval workflows for more automated QA -Versioning & user change logs coming very soon And agree the company that can becomes the repository of the majority of 3D assets is well suited to win the 3D AI space.

23

Portkey is now a first class provider in the Vercel AI SDK. When should you use Portkey's AI gateway with the AI SDK? 1. You're routing requests to multiple models & providers. 2. You're going to production and need metrics on cost, performance and accuracy 3. You want to build & manage on-line guardrails without adding latency 4. You have multiple teams building with AI and need to manage usage and budgets.

57

1 Comment

#ObsessivelyEnthusiasticInvestor #UnreasonablyEnthusiasticInvestor #IrrationallyEnthusiasticInvestor at REFASHIOND Ventures: The Industrial Transformation Fund. We look for the people we believe are doing the best work in Industrial Transformation & Supply Chain Technology; Attempting the impossible. Then we go all-in on them. With obsessive and unreasonable enthusiasm. Join us. #IndustrialTransformation #SupplyChain #Innovation #Technology #Startups #VentureCapital #DemandTheImpossible #UnreasonableEnthusiasm #ObsessiveEnthusiasm #IrrationalEnthusiasm 

6

I just took a look at Meta's Llama 3.1 announcement. While everyone’s talking about the latest benchmarks, I have some interesting takeaways here. 1. They’re now letting developers use Llama’s output to improve other models. It’s pretty hard to understate how huge this is - generating synthetic data unlocks a lot of new capabilities for anyone working on smaller models. 2. In that same vein, synthetic data played a huge role in Llama 3.1’s improvement. A common sentiment in data science is that data is the biggest bottleneck - the improvements that Meta’s achieved suggests this might not be as important as we thought. 3. I think we’re seeing the emergence of a new OS. Andrej Karpathy has already talked about this - an “LLM OS” with applications built around it. Imagine if the future of software is on an LLM OS, and Llama becomes its Linux - true, top-quality, maintained open-source? That would be huge.

59

5 Comments

Llama-3 may have just killed proprietary AI models !!   Meta just put out Llama-3 few days ago, but it already feels like the moment when open-source models finally caught up to private models. The first tests show that Llama-3 70B is pretty close to GPT-4 in a lot of areas: - Artificial Analysis reveals that Llama-3's quality is in between that of Gemini-1.5 and Opus/GPT-4. Please see: https://lnkd.in/d8QGHNzh - The official Meta page only lists Llama-3 as better than Gemini 1.5 and Claude Sonnet. Please see: https://lnkd.in/dzJ9xzJM - Llama-3 is placed #5 on the LMSYS Chatbot Arena Leaderboard. Current GPT-4 models and Claude Opus are still tied for first place. Please see: https://arena.lmsys.org/ The even stronger Llama-3 400B+ model is still being trained, but it will probably be stronger than the GPT-4 and Opus when it comes out. OpeningAI vs. Meta --- has Meta really won? Many people think that Meta's original goal was to "scorched earth" OpenAI by putting out strong open models to change the competition and make sure they didn't fall behind in the AI race. And it seems to be working as: - Meta can probably spend more than OpenAI on skill and computers. - OpenAI is thought to make about $2B a year, but it probably isn't making any money. In 2023, Meta made $134B in sales and $39B in income. - By now, Meta's computing power probably beats OpenAI's. - It's likely that open source brings in better experts and workers. In order to catch up with Meta, Microsoft might buy OpenAI as one alternative. Google is also moving into the open model space, and its tools are a lot like Meta's. It will be interesting to see how they fit in. But the real winners are the Developers and new AI product companies. Now that Llama-3 is out, companies have even more chances to make money: a. Your product gets better automatically as each big model improvement is made b. Developers don't have to wrap private API endpoints in AI anymore; they can now add AI deeply to their products in a way that is both cost-effective and fast. Hugging Face already has more than 800 llama-3 variations, so it looks like everyone will be able to make it work for their need, language, or business. c. Hardware that works faster and costs less: Groq can now make 800 llama-3 tokens per second for a small part of what GPT does. Soon, LLM answers will be almost instant and cheap. But open-source mixed models for video and vision still need to catch up. I think this will happen very soon. The release of Llama-3 is a big step towards making AI more accessible to everyone, but it's probably too early to say that private models are dead. There's a chance that GPT-5 will surprise us all and do things we never thought transformer models could do.   #Meta #LLAMA3 #ChatGPT4 #groq #claude #opus #OpenAI #Google #Microsoft #artificialintelligence #machinelearning #opensource

1

1 Comment

People rightly state that there is a lot of hype surrounding #AI. This reflects our obsession with applying new tools to *every* problem, fit for purpose or not. 😄 Here's the good part, though: when applied properly, AI is a huge force multiplier. Imagine a lever ^ 1000000. I've been sharing, any time I have the opportunity, the "good parts": areas in which AI can have real utility now. Not someday. Not maybe. Absolutely yes, and now. One area in particular is #RAG, Retrieval Augmented Generation. RAG == Turning an AI loose on YOUR DATA. Imagine all of those documents in which vital information is squirreled away, hard to search, even harder to correlate *across* those myriad documents. What a treasure trove if only you could leverage it! With RAG, you can...and pretty easily, too. Using AI and RAG, 1+1 pretty quickly adds up to 500. Or more. Even now, the power of your AI-enabled application is more limited by your imagination than the capabilities of the tools. If you want to know more, get just a taste of what you can do with AI using RAG, join me this Monday at #RAGHack. See below for more details and to save your spot! https://lnkd.in/gYAsiSP4

13

1 Comment

When we started evaluating which LLMs to use in our product at SpotDraft, it wasn’t simply a question of “Which model performs best?” The challenge was much bigger. We needed an LLM that could adapt, scale, and actually add value to our workflows, not just score well on benchmarks. We tested OpenAI, Google and Anthropic models simulating high-stakes conditions to see where they excel—and where they hit bottlenecks. We looked beyond accuracy or latency; it was about real-world utility and resilience. Here’s what we uncovered: Key factors like response times under load, token efficiency, and contextual retention drove our decisions. For instance, one model excelled in structured data tasks but lagged in nuanced legal prompts. Another model excelled in giving nuanced answers but took a long time to share the output. Each result informed how we fine-tune model selection, optimize pipelines, and adapt architecture. We had to evaluate alignment with both immediate needs and scalability over time. Takeaways for any engineering team diving into LLMs: - Test models against real workflows, not just ideal scenarios. - Evaluate trade offs for each model: context length vs time to execute vs accuracy vs cost. - Select models that allow for domain-specific adaptations with minimal retraining. - Build in a model-agnostic way so you can change course as and when needed. It’s not about picking the “best” model; it’s about building a sustainable framework that evolves with your needs. How are you going about evaluating LLMs?

151

9 Comments

Meta just dropped Llama 3 (70b) last week. Here's what you need to know. Trained on 15 trillion tokens of data. 😯 Llama 3 is leading the space in capabilities for math and reasoning. Scoring an 85 on the MMLU benchmark. (Massive Multitask Language Understanding) Compared to Claude 3's impressive 86.9 score. However, with 25 trillion less training tokens. Llama 3 definitely has the potential to exceed Claude 3. The most exciting development comes in the form of Meta's approach to AI. 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲. Llama 3's being released to developers, Along with a "Getting Started" guide, as well as new trust/safety features. - Llama Guard 2 A beefier version of their Llama guard content classification system. - Code Shield A screener for potentially malicious, or improper code outputs. - CyberSec Eval 2 A system for protection against prompt injections and code interpreter abuse. 💡 Benefits of Open Sourcing Llama 3 Llama 3 is comparable to GPT-4 in terms of capabilities. Tech hardware companies like Nvidia and AMD are supporting the model, As well as Cloud services like AWS, and Hugging Face. By it becoming publicly available to whomever wants to design with it. ✅We'll quickly have many GPT-4 models entering the market. The competition is on! - I'm really excited to announce Hank and I's collaboration with Christoph Rosenboom Keep an eye out next Monday! #llama3 #AI #LargeLanguageModel

17

19 Comments