Kurt Smith

Databricks disclosed that they are expecting to cross $2.4B in revenue run-rate growing ~65% Y/Y by H1’25 (Jan FYE). This was alongside top-tier metrics, including 80%+ GMs, 140%+ NDR, and a multi-product story where ~50% of customers use 6+ products (incl. their Data Warehouse product which generated $400M in revenue up 4.0x Y/Y). Given their accelerated growth, Databricks is among the few scaled software companies, alongside hyperscalers, capitalizing on the AI boom. Equally noteworthy is their market share gain against competitors. Considering their growth trajectory, we explored various scenarios to determine when Databricks will surpass Snowflake in scale. The question is not if, but when, and we believe it is sooner than most anticipate.

1,010

26 Comments

This new little product demo is awesome! Our new Rev AI Transcript Assistant will save you TONS of time. Easily pull out nuggets of info from your transcripts, making valuable content more accessible than ever. https://lnkd.in/gzzFpcDa #transcription #asr #accessibility #transcript #summarization #ai #assistant

31

2 Comments

❄ In case you missed it, Snowflake recently released Arctic, a family of performant and compute-efficient Apache 2.0 licensed Large Language Models (LLMs). Curious about how Arctic might fit your needs? Come explore its capabilities at Clarifai alongside the 70+ other LLMs currently available on our platform. Let's build the next groundbreaking GenAI app together! https://lnkd.in/gNz5Babr

20

There is often an unspoken assumption by ML researchers that all we need to make progress and model every aspect of our world is data. We do not ascribe to this in my lab. In fact, we believe that data will never be enough. Our experiences with the nuances and complexities of scientific problems have informed us to the insufficiency of data to capture continuous physical processes, which afterall govern our biological and physical world. An example of this is this series of two papers led by my wonderful PhD student, Anowarul Kabir and advanced by a precious multi-year collaboration of my lab with Los Alamos National Lab: Anowarul Kabir, Manish Bhattarai, Kim Rasmussen, Amarda Shehu, Anny Usheva, Alan R Bishop, and Boian S Alexandrov. Examining DNA Breathing with pyDNA-EPBD. Bioinformatics 39(11):btad699, 2023. https://lnkd.in/gZaHE4hS Anowarul Kabir, Manish Bhattarai, Selma Peterson, Yonatan Najman-Licht, Kim Ø Rasmussen, Amarda Shehu, Alan R Bishop, Boian Alexandrov, Anny Usheva. DNA Breathing Integration with Deep learning Foundational Model Advances Genome-wide Binding Prediction of Human Transcription Factors. Nucleic Acids Research: gkae783, 2024. https://lnkd.in/gfTTqcPy Our goal: advance an exceptionally challenging problem in molecular biology, prediction of transcription factor binding sites. Our first step: capture the underlying physics that is missing in the data. Our second step: integrate that now with the data we have in a foundation model for predicting transcription factor binding sites. Performance improves. Most importantly, when we look at the sequence motifs that constitute "signatures" of what makes a transcription factor binding site, we obtain answers. All in the open, nothing opaque.

33

1 Comment

🤔 Ready to build smarter, more efficient agents? My latest blog post dives into the world of smaller, task-specific LLMs designed to power the next generation of agent-based systems. 💪 Discover how these models enhance agent capabilities while minimizing resource consumption. #AI #LLM #AgentBasedSystems #DeepMind

38

3 Comments

108 Recipies Snowflake Build: Just moments ago (Nov 12th, 12:30 PM CST), SF presented the Snowflake Dynamic Iceberg table using Polaris External CatalogSync." We have 20+ recipes in this e-book. Below is how to build summary tables as new data streaming into the base table. Snowflake

6

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

There is so much research going into bringing the power of LLMs into the Enterprise context. The main techniques are k-shot learning or prompt engineering, RAG (Retrieval Augmented Generation) & Model Fine tuning (not to mention many variations of these e.g. Agentic RAG, GraphRAG etc.) However, I came across an interesting take on this from IBM and they are calling it "InstructLab" - where the claim is that they are able to teach the LLM "new skills" grounded in an Enterprise context. On the surface it looks like a k-shot learning + fine turning on steroids. Approach seems to be to utilize a small "skills training dataset" in order to generate a large synthetic dataset (perhaps utilizing a specialized LLM) that is then incorporated into the LLM weights natively. This could be a very interesting concept as long as the cost of adding new skills is not very high (when compared to fine tuning). Exciting times! #genAI #enterpriseAI

40

2 Comments

Exciting news! Snowflake has unveiled a cutting-edge system for large language models, featuring the powerful Llama 3.1 405B from Meta. This innovation brings a new level of efficiency and capability to AI applications. At 7Rivers, we're excited to harness this technology, enhancing our Data-Native™ approach to deliver unparalleled AI solutions for our clients. Our commitment at 7Rivers is to drive business value through use of intelligent data and advanced AI. Snowflake's latest offering, which allows real-time AI processing with minimal infrastructure, perfectly complements our mission. This advancement enables us to provide our clients with scalable, cost-effective AI applications, enhancing operational efficiency and driving impactful outcomes. The integration of Meta’s Llama 3.1 405B within Snowflake Cortex AI aligns with our Data Native™ Model, empowering us to create transformative solutions, optimizing processes, and creating a competitive advantage. Together with Snowflake, we're revolutionizing data usage to create intelligent, augmented business experiences. #AI #DataNative #7Rivers #Snowflake #GenerativeAI #BusinessTransformation

34

2 Comments

Open has become the state-of-the-art (for now), Meta have changed the game - yet again - with the release of the Llama3.1 suite of models. The acceleration in this industry is breathtaking. The headlines: * Llama 3.1 models are immediately available in Cortex today. ✅ * 405B packs a punch - out performing other leading flagship models across a variety of benchmarks. 📈 * 8b & 70b models have been updated too. The already capable models have seen significant jumps, particularly in math and coding, in just 3 months. 🚀 * Meta's 3.1 suite will be made available for serverless fine tuning. 🤯 * We've also released Snowflake Cortex Guard (GA) to safeguard LLM applications. 🛡

8

3 Comments

💪 Super excited to finally share the new Spark Performance Optimization capabilities on the definity platform! ✨ Spark data engineering teams and platform leaders -- ready to unlock performance insights and cost savings from day-1 ? ⚡ Proud of the definity team who's moving at lightning speed to deliver tangible value to our customers. Check out the article by Tom Bar-Yacov (link below) and reach out to us to learn more!

32

3 Comments

1. IMO, agent patterns are one of the most interesting ways to extend LLMs. memary is based on leveraging a graph database (neo4j) for long-term memory (to get around the limited memory inherent in even the largest context windows in LLMs) and augmenting it with other tools like web search and Computer Vision. https://lnkd.in/gfFcBqJx Looks interesting. 2. More about how consolidation effects the entire healthcare industry: https://lnkd.in/gXJKqNJB The Change hack/ransomware attack not only affected claims, service, refills, etc. but now CMS & NCQA are delaying the entire HEDIS reporting process (used for Medicare/Medicaid quality ratings) due to the impact. 3. NARA bans ChatGPT use agency-wide. https://lnkd.in/gW8T94JU Federal Government agencies see this as a risk, including their information leaking and being used to train models (which may then output the wrong information or show PII/PHI other proprietary information). 4. Oh great, now I can't even be angry anymore :) https://lnkd.in/gcrN9mjx

1

1 Comment

I really enjoyed my conversation with Bryan Wood and Jeremy Kuhnash from RedPanda at the Databricks conference! I remember bumping into Redpanda when I was researching and writing my ebook for Redis (Redis Streams vs. Kafka - a visual guide) nearly 2 years ago! https://lnkd.in/ga4vbvaN For those who don't know, RedPanda is a faster, a more cost-effective, and a drop-in replacement to Kafka. For example: say for some workloads, you may need 10 Kafka brokers, with RedPanda, you'd need just 3 - that's a huge savings! So if you are thinking of using Kafka or Confluent, you should definitely check them out! https://redpanda.com #kafka #confluent #streaming Databricks

35

Given my experience with LangChain, building PDF document chunks and generating embeddings through the Palantir Build Pipeline is straightforward. Before you learn this tool, learn the following: Chunks, Embeddings, VectorDB, Similarity Search, Ranking, Prompt, etc in LangChain. Of course, Pyhton is your AK47 Palantir Technologies

10

SEAL is a new model benchmarking service from Scale AI that fills a big gap: an additional objective, third-party, trustworthy way to compare models. People distrust many benchmarks because they worry that models are optimized to perform well on them—including by training on the underlying data (like cheating on a test by memorizing the questions and answers in advance). The LMSys Chatbot Arena (https://chat.lmsys.org) addresses this by having humans evaluate model output in blinded head-to-head tests. The new SEAL leaderboard (https://lnkd.in/gdCZzRnz) does something similar, but uses "thoroughly vetted experts using domain specific methodologies" to do the head-to-head evaluation. Sounds much more expensive, but they just raised $1 billion, so I think they can afford it.

1

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

Are you maximizing the full value from your data? Over the past year, Gradient has developed our AI-powered Data Reasoning Platform that enables businesses to automate complex data workflows.  https://lnkd.in/gqVmHMdS Specifically, Gradient is helping businesses: ✅ Leverage the full capacity of their data, especially the institutional knowledge that tends to go untouched due to its unstructured nature. ✅ Run complex data workflows that not only extract data but also apply a higher order of operations to it that requires reasoning. Deep dive with us as we break down the challenges of working with complex data and performing logical reasoning reliably, even with a fully staffed team.

42

1 Comment

Using LLMs for navigating publicly available healthcare datasets I recently found out that Databricks uses OpenAI for its Databricks Assistant function. It's funny that, as I was playing around with Databricks Assistant, it revealed itself, saying, "I'm sorry for any confusion, but as an AI developed by OpenAI..." As a big fan of Claude.ai (by Anthropic), I decided to create another agent powered by Claude and put it on my Slack group. I worked on that yesterday and this morning. The video below is the result of the work. So far, it is able to do what I expected it to do, but I think it can do a lot better. If you guys have any tips on making LLMs smarter in a specialized context, please help me! Large Language Models (LLMs) will forever change how we interact with complex data. They can help us find relevant datasets from the oceans of data, help us write queries, help us combine them, and help us come up with new and important research questions that we never thought of. It's a truly exciting time. https://lnkd.in/e4RiYNrk #opendata #healthcaredata #healthcareanalytics #healthcaretechnology

53

3 Comments