Scott Juang

Lately, we have observed a transition from Cloud Databases(Snowflake etc) to Iceberg OpenLakehouse format. Ambaricloud team is working on Snowflake to Iceberg Automation and end-user training. Why: Low-cost computing. SF is still a favored engine for Gold layer data. Less stickiness. More engine options to run on Iceberg tables. Cloud storage Parquet files Iceberg table format Catalog Status: Polaris -- Started Walking. Hopefully, it will Jog or Run soon! UnitCatalog -- Infant Glue -- Safe bet for now. Comments? Snowflake Cloudera Dremio Upsolver Databricks

8

1 Comment

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

9

These features help HPC users save 50-80% on EC2 compute costs...here's how they work -- Optimize is a feature we developed for stateful jobs running on EC2 On-Demand and migrates them to run on EC2 Spot instances: 1. This compute optimization feature uses our memory checkpointing technology to ensure the job doesn't lose state This is key because for long-running scientific computing jobs it is difficult to restart what could be hours, days or even weeks of runtime just to save $ 2. During extremely busy periods, EC2 Spot interruption rates can be as high as 50-80% and responding to each rebalance notification is very disruptive If you're running on a Spot instance that actually get reclaimed and our software can't easily find another Spot instance, we perform a stateful migration to on-demand instances and Optimize helps look for a Spot instance 10, 20, 30 mins later to move back to -- How many times have you run out of memory? Out of memory protection is a feature we developed for HPC jobs that are hard to deterministically predict memory requirements pre-run: 1. Being able to run on smaller EC2 instances and migrating to larger instances only when needed without losing state results in material cost savings 2. Our software monitors resource utilization on all the worker nodes and when we detect memory utilization approaching 100% and/or swap is hit, we perform a full application checkpoint, migrate you to the larger instance, and resume running -- Some of these capabilities that we have developed to optimize Cloud costs are also quite useful for on-prem HPC application. Drop a comment below if you have questions or shoot me a DM if you want to chat on how implement memory checkpointing in your HPC architecture. #AWS #HPC #SuperComputing #Slurm #PEARC24

11

2 Comments

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

4

Speed is a competitive advantage, and Snowflake proves it every day! Couldn't be prouder of the nonstop innovations and the direct impact they're making for our customers. The era of Enterprise AI is here—and Snowflake is leading the charge! #Snowflake #Innovation #EnterpriseAI #SnowflakeIntelligence #AIDataCloud

3

Same Chicken (Iceberg), different Curries (Engines) Cloudera: Read Snowflake Data from Polaris External Sync Catalog I am able to "read" and integrate Snowflake data through Polaris with other data in Cloudera Spark. Cloudera Snowflake

17

✨ If you have Databricks, do you need Snorkel AI? If you plan to customize #LLMs with your own enterprise data the answer is an easy yes. You need high quality data to build production-quality AI, but most unstructured enterprise data is inaccurate, incomplete, and inconsistent. Often you have a lot of data for common scenarios but very little for high-value edge cases (obviously!). Good news: Snorkel Flow integrates seamlessly with Databricks at every stage of the AI development workflow, from accessing data & models to deploying production-quality applications via Unity Catalog. For a guided walk-through check out this 5min demo and learn more at https://lnkd.in/gTiQ4wD8 #AIInnovation #ArtificialIntelligence #dataintelligence

31

7 Comments

Databricks teamed up with AWS to power customized, production-quality #GenAI applications through AWS Trainium AI chips! The collaboration builds on Databricks’ newly expanded #MosaicAI capabilities — including Mosaic AI Model Serving, which supports a wide variety of model providers. Together, Databricks + Amazon Web Services (AWS) will make it possible for thousands of joint customers to build, deploy and monitor high-quality, customized #AI applications without sacrificing control over their data or IP.

2

Really interesting read on the evolution of Data Warehouses, Data Lakes and Lake Houses and how Databricks and Snowflake are battling for the top spot. https://lnkd.in/egg4P5VG This is one of the more complete analysis I've seen on the evolution of the analytics space and it's big gorilla's. In the end, it gets into the AI discussion of big LLM's vs. small models and how SLM's will have a place in the analytics world, which I completely agree with.

9

1 Comment

Databricks disclosed that they are expecting to cross $2.4B in revenue run-rate growing ~65% Y/Y by H1’25 (Jan FYE). This was alongside top-tier metrics, including 80%+ GMs, 140%+ NDR, and a multi-product story where ~50% of customers use 6+ products (incl. their Data Warehouse product which generated $400M in revenue up 4.0x Y/Y). Given their accelerated growth, Databricks is among the few scaled software companies, alongside hyperscalers, capitalizing on the AI boom. Equally noteworthy is their market share gain against competitors. Considering their growth trajectory, we explored various scenarios to determine when Databricks will surpass Snowflake in scale. The question is not if, but when, and we believe it is sooner than most anticipate.

1,010

26 Comments

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

7

Open has become the state-of-the-art (for now), Meta have changed the game - yet again - with the release of the Llama3.1 suite of models. The acceleration in this industry is breathtaking. The headlines: * Llama 3.1 models are immediately available in Cortex today. ✅ * 405B packs a punch - out performing other leading flagship models across a variety of benchmarks. 📈 * 8b & 70b models have been updated too. The already capable models have seen significant jumps, particularly in math and coding, in just 3 months. 🚀 * Meta's 3.1 suite will be made available for serverless fine tuning. 🤯 * We've also released Snowflake Cortex Guard (GA) to safeguard LLM applications. 🛡

8

3 Comments

❄️ Come and hear how Snowflake uses DataOps.live for massive productivity gains for all its engineers. #data #dataops #analytics #dataproducts

10

Hey Databricks peeps!🐥 Mark down May 15th for a Data Modeling Webinar with Keith Belanger & Siddhesh Pore! Don’t miss this opportunity to learn: 🔺 Fresh updates for the Databricks SQL serverless warehouse 🔺What is data modeling and why it matters 🔺6 steps to data modeling Excellence in medallion architecture 🔺The benefits of upgrading to Unity Catalog 🔺How to leverage proactive model-centric Data Governance 🔺Live demo: Building a data model for Databricks SQL 🔺Q&A #SqlDBM #datamodeling #databricks https://lnkd.in/gXThVWsd

5

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

3