Russell Williams

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Comments

💡 Optimizing Costs with Dragonfly: A Technical Breakdown 💻 For technical teams looking to maximize performance while minimizing cloud infrastructure costs, Dragonfly offers significant advantages. By cutting down memory overhead, leveraging CPU-specific optimizations, and using a shared-nothing architecture, Dragonfly is designed to be resource-efficient while maintaining high throughput. Whether you’re scaling horizontally or aiming to optimize performance, Dragonfly can help you reduce operational costs without sacrificing speed. In this blog, we break down the technical strategies that Dragonfly uses to cut costs and improve efficiency. 🔗 Read more here: https://hubs.la/Q02RDNVr0 #CloudOptimization #TechEfficiency #DevOps #DatabasePerformance #Dragonfly#CostReduction

3

Want to know more about the future of Delta Lake and Apache Iceberg? We had the chance to interview Tathagata Das alias TD, major contributor to Delta Lake, who shared more insights on what's new & the direction both formats are taking. And he also shared some interesting metrics - counting in Baby Shark units - We discussed the past, present and future of Delta Lake, including the major features we all love - Clustering and the new Variant type! Watch the full episode now! ==> https://lnkd.in/e9wrTFsm Thanks Youssef Mrini & TD!

46

2 Comments

👋 🛠️ Say goodbye to complex troubleshooting. With Synthia, the AI assistant built into Sisense's Compose SDK, developers get real-time guidance—no technical jargon needed. Whether you’re a seasoned coder or new to the game, Synthia makes navigating Compose SDK seamless, solving challenges and optimizing workflows faster. Start your 30-day free trial 👉 https://bit.ly/4dGzK7j #AI #SaaS #DataScience

30

Step by (2nd) step using Apache Hop for ETL building and deploying

5

The final deep dive on change query support in the table formats is ready. This one covers Apache Paimon, focusing on understanding the change query support through the lens of its internals. https://lnkd.in/dFfM_8nA

28

OpenLogic by Perforce is collecting responses for our 2025 State of Open Source Report, which will explore industry adoption trends, license changes, compliance, and more. I encourage all folks working with OSS to take a few minutes to tell us what technologies your organization is using and why. Here's a link to the survey: https://lnkd.in/gS_h-xYh Thanks in advance for participating and helping us put out a great report! #oss

9

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

If you're interested in learning more about Amazon Q generative SQL capabilities and seeing them in action, check out this video link. A big thanks to Xiao Qin for putting this together on AWS OnAir! https://lnkd.in/d2udk9bt

29

new post - we chatted with Davis Liang) about some of the engineering challenges and breakthroughs for a company training their own healthcare LLMs -dealing with multiple languages -handling model drift -generalist models vs. healthcare specific ones and more. You can read it here: https://lnkd.in/dgmZY35G (just fyi, this one is a sponsored post but it's pretty interesting if you're interested in healthcare-specific LLMs)

35

1 Comment

The subject of support for incremental reads of mutable tables and generating CDC streams in the four table formats I'm covering is far more nuanced than I expected. What was going to be one post is going to have to be a 5 parter to keep it manageable: 1. Introduction to the concepts 2-5. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon

15

2 Comments

Learn how to power real-time data streaming applications with Couchbase Capella and Amazon MSK in our latest blog post

3

Real-Time #Analytics News from the Past Week: 1. Starburst announced a range of new capabilities for their Trino-based open hybrid lakehouse platform, Galaxy, including: -The general availability of fully managed streaming ingestion from #ApacheKafka to Apache Iceberg tables. -The public preview of fully managed ingestion from files landing in Amazon Web Services (AWS) S3 to Iceberg tables. -Multiple enhancements to performance and price-performance of their #lakehouse platform. 2. Cohesity introduced a patent-pending visual data exploration capability to Cohesity Gaia, its #AI-powered search assistant launched earlier this year. By providing customers with a visual categorization of the themes across documents and files within a #data set, the visual data explorer brings new context to the data and suggests queries that help users gain insights faster. 3. IBM announced the release of its most advanced family of AI models to date, Granite 3.0. IBM’s third-generation Granite flagship language models can outperform or match similarly sized models from leading model providers on many academic and industry benchmarks. 4. Confluent announced the launch of the Confluent for Startups AI Accelerator Program. The 10-week virtual program seeks to collaborate with 10 to 15 early-stage AI startups that are building real-time applications utilizing the power of #DataStreaming. 5. Semantic Web Company (SWC) and Ontotext announced that the two companies have merged to become the Graph AI provider, Graphwise. Read the rest on RTInsights ➡️ https://lnkd.in/eQKsrPCZ

12

You don’t need to build complex tests to build performance comparisons. This is by far the most practical tests I ve come across that explains Databricks seemingly cost effective and performant in practical use cases.

3

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

20

Using LLM in #snowflake cortex with OCR in snowpark to play video games! #DataArchictectGoneWild https://lnkd.in/gAQnmXdn

1