Michael Armbrust

The subject of support for incremental reads of mutable tables and generating CDC streams in the four table formats I'm covering is far more nuanced than I expected. What was going to be one post is going to have to be a 5 parter to keep it manageable: 1. Introduction to the concepts 2-5. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon

15

2 Comments

Substack recently moved their ML and AI pipelines over to Modal, from AWS SageMaker. We've had a lot of fun working with them and we just wrote about it on our blog. “Using SageMaker just felt like a very convoluted process. It felt very removed from a normal engineering workflow. With Modal, it’s a lot faster. You don’t feel like you have to wait for notebooks to spin up or remember to turn them off or any of that type of stuff.” – quote from Mike Cohen, Head of AI & ML Engineering at Substack In the process, we actually made several improvements to Modal's system latency, since Substack's use case is very latency-sensitive. These optimizations now benefit all Modal users. Take a look: https://lnkd.in/e8yMCiti

185

5 Comments

I just updated my first table format comparison blog post to fix some missing details of Apache Hudi. https://lnkd.in/dwSUS6qS

5

You don’t need to build complex tests to build performance comparisons. This is by far the most practical tests I ve come across that explains Databricks seemingly cost effective and performant in practical use cases.

3

Just over a week away from the MLOps Community webinar on Agents in Production! 🗣️ You can catch my session on Generative AI Agents in Prod from 12:55p - 1:20p CST. Everyone knows how to build an Agent by this point (right?) 😅 I'm going to skip all the basics and discuss lessons learned and "gotchas" from spending the last couple of years deploying Production quality Agents. Building demos is easy. Quality and Reliability in Production is hard. Let's talk about it. See you all next week! https://lnkd.in/gf553FMi #AgentsInProd #Gemini #Google #GoogleCloud #VertexAI #GenAI

102

3 Comments

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

Awesome work by the StrongDM team to get use to 1.0.0 release for Cedar Go! Why Cedar-Go is a game-changer: ✅ Handles large-scale policies with unmatched performance (250k+ resources in <200ms). ✅ Simplifies policy creation with intuitive programmatic AST construction in Go. ✅ Accelerates batch authorization for complex datasets. ✅ Built for trust with 100% test coverage. Check out some sample policies here: https://lnkd.in/gmcksWvN

17

Now we have two first class data catalogs integrated with MS Fabric now available in Azure. Of course DataBricks is the heavy heavy favorite in the enterprise lakehouses as well as AI (as of the conference that just finished 🤓)

6

1 Comment

Hey folks! I'd like to more formally introduce and talk about a few of the projects we're working on towards improving large-scale bioinformatics analyses (one of which was developed during St. Jude's KIDS24 Biohackathon)! If you're not aware, my team started a project called St. Jude Rust Labs (https://lnkd.in/eXY_9dJ8) where we're rewriting a more foundation for bioinformatics analysis in Rust. This cuts across more than just workflow execution, but, recently, we've been primarily focused on improving the experience of working with the Workflow Definition Language (WDL) at scale. To that end, we've released a number of projects, including: - We've written a complete lexer/parser for WDL v1.2 language (https://lnkd.in/eGWHSM2g). Anyone can use this foundation to build tools for WDL. - We've written a VSCode plugin that includes a language server protocol (LSP) for linting and validation directly in your editor (https://lnkd.in/euApG9tF). - And, announcing today, we've started down the road of writing our own WDL execution engine spread across the Crankshaft (https://lnkd.in/et_RFkA3) and Sprocket (https://lnkd.in/eUTuPE68) projects. Crankshaft was prototyped during the St. Jude Biohackathon last week—it's a _headless_ workflow execution engine, meaning that, in theory, others could come along and write drivers for Crankshaft built on NextFlow, CWL, Snakemake, etc. As I said earlier, our team is really focused on WDL specifically, so we're going to continue building out a "head unit" for Crankshaft using WDL. That being said, I would love to see other community projects popping up and using the core machinery for other workflow languages. Thanks to all of the individuals who participated on our Biohackathon team (Kevin Benton, suchitra chavan, Braden Everson, Andrew Frantz, Michael Gattas, Peter Huene, and John McGuigan)!

76

1 Comment

First blog post in a long time! I started writing a post ~2 years ago on adventures counting 130M U.S. buildings by zipcode and finally circled back to write it up properly. Everybody is a winner really, but DuckDB, Ibis, Apache Arrow, and GeoParquet were absolutely essential for getting the data into and out of all of the various engines. Give it a read and let me know what I missed! https://lnkd.in/evgVe3vD

60

2 Comments

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

2 Comments

A new table-format comparison post is out. How do the table formats support the streaming ingestion of row-level operations (insert/update/delete), such as ingesting CDC streams? https://lnkd.in/dFDs4Ah3

36

There's been a growing focus on effective benchmarking methods for large language models. Came across an interesting study introducing a new approach that utilizes LLM debates to assess model performance. The method involves creating an LLM debate framework and using an LLM judge that analyzes debate arguments based on criteria like clarity, factual accuracy, counter-arguments, and persuasiveness. This eliminates the need for human evaluation, making it more scalable for LLM development and evaluation. Could this be the step towards LLMs acting as judges in the future? Are we ready for AI referees? 🙂 #AI #machinelearning #LLMs #naturallanguageprocessing https://lnkd.in/g7A8pEUW

14

2 Comments

🎃The October issue of #CheckpointChronicle is now out 🌟 It covers Ververica's Fluss, #ApacheFlink 2.0, Iggy.rs, Strimzi's support for #ApacheKafka 4.0, tons of OTF material from Jack Vanlightly, Christian Hollinger's write up of ngrok's data platform, nice detail of how SmartNews use #ApacheIceberg with Flink and #ApacheSpark, a good writeup from Sudhendu P. on #ApachePolaris, notes from Kir Titievsky 🇺🇦 on Kafka's Avro serialisers, and much more! https://lnkd.in/djc-HERy

18

My blog post about what I’ve been working on at Rippling the last year and a bit is live!  Come learn how Rippling Engineering is building data residency for the EU and beyond https://lnkd.in/gy-2ebD7

64

1 Comment

Want to run #llama models on #arm devices? Check out the latest in llm’s on arm devices. #xnnpack #android #chatbot #meta #exucutorch #pytorch

6

I just published my latest blog post: "Experimenting with Rust, Nix, Parca & k6" https://lnkd.in/eUKeA57g This one is a long one, but it turned out well I think! #nix #rust #parca #blog

25

3 Comments