Stefano Fago’s Post

AutoMQ will surely bring you gains in the innovation of building stream storage engines based on the cloud. If you have doubts about this, why not try it out for yourself? It's worth mentioning that the source code of AutoMQ is open on Github, and you can deploy it to your production environment for free. #warpstream #apachekafka #confluent #streaming

Have you worked with WarpStream? Deployed to / running, serving in production? Would like to hear your experience, comments on it esp. compared to Kafka. About WarpStream from their website: "WarpStream is an Apache Kafka compatible data streaming platform built directly on top of object storage: no inter-AZ networking costs, no disks to manage, and infinitely scalable, all within your VPC. → Zero Disks to Manage → 10x Cheaper than Kafka WarpStream is deployed as a stateless and auto scaling agent binary in your VPC with no local disks to manage." https://lnkd.in/gVdYW23x

Kafka has been there for quite some time. Everybody and their uncle uses it for data streaming and processing. You know it can perform, and we know it. But how do you get the most bang for your buck out of it? We rigorously tested Kafka performance, comparing different environments to find the most cost-effective setup in AWS or GCP. We ran dozens of tests to find which compute generation, CPU, and JVM would yield the best price-performance ratio for Kafka. Specifically, we measured how many millions of rows we can ingest into the Kafka broker per one cent. Read more in our blog (spoiler: ARM rocks): https://lnkd.in/eJ7aRpbw

Congratulations! 🎉 🎉 Our open-source project AutoMQ, after making it to GitHub Trending on 6.22, is once again on GitHub Trending today. We appreciate the recognition from the developer community. AutoMQ, as a Cloud First streaming system, is attracting a lot of attention from developers in the data streaming field. We have redesigned the storage layer of Apache Kafka based on EBS WAL and AWS S3. While ensuring 100% compatibility with Apache Kafka, AutoMQ offers over 10x cost efficiency and over 100x partition migration efficiency compared to Apache Kafka. This allows your streaming system to scale freely in seconds, significantly reducing the complexity of using streaming systems. For more information, please refer to: AutoMQ Github Repository: https://lnkd.in/g8Gh9U9m Github Trending Page: https://lnkd.in/eM6Uqvw8

Really excited to be working with my friend and fellow committer Jordan West on a performance patch for Apache Cassandra that's already showing a MASSIVE 6x reduction in IOPS usage during compaction which makes a huge impact on EBS. In our initial testing we saw a 3x improvement to compaction throughput, just on one thread, and multiple threads will deliver an even bigger improvement. The same patch will reduce system overhead on *any* deployment, but will have the biggest impact when using disks that use quotas on IOPs. The impact is you'll be able to run WAY denser nodes on EBS for the same cost, more predictable random read performance, and we're not even finished. This same patch should deliver improvements to range scans (hello spark jobs!) and repairs. You can follow along here: https://lnkd.in/d5_hcyJQ

With Confluent Platform 7.8, stream processing with Apache Flink is even easier on-prem and private clouds! 🐿️ Learn more about Confluent Manager for Apache Flink and the other 🆕 goodness included in this release on the Confluent blog:

🚀 Unlocking the Power of Redis: A Game-Changer for Data Management 🚀 In today’s fast-paced digital world, efficient data management is crucial for delivering high-performance applications. That’s where Redis comes in! 🌟 Redis, an open-source, in-memory data structure store, is renowned for its speed and versatility. Whether you’re dealing with caching, real-time analytics, or message brokering, Redis has got you covered. Here are a few reasons why Redis should be on your radar: 🔵 Blazing Fast Performance: Redis operates in-memory, which means data access is lightning-fast. This makes it ideal for applications requiring quick response times. 🔵 Versatile Data Structures: From strings and hashes to lists and sets, Redis supports a variety of data structures, allowing you to choose the best fit for your use case. 🔵 Scalability: Redis can handle millions of requests per second with ease, making it perfect for scaling applications. 🔵 Rich Feature Set: With features like persistence, replication, and Lua scripting, Redis offers robust tools to enhance your application’s capabilities. Curious to learn more? Dive into the world of Redis and see how it can transform your data management strategy! 🌐🔍 https://lnkd.in/dqzvfVnF #Redis #DataManagement #TechInnovation #HighPerformance #Scalability

Storage and compute will be split until morale improves. This time another Kafka implementation named Automq, claiming 10x compute and 12x storage cost savings by using blob storage. So far I only knew about Warpstream, but competition intensifies, which is always good.

While reading about building scalable data systems, i came across few tools with dual functionality- Streaming Platforms with Storage: - Kafka: Kafka isn’t just for real-time data streaming, it also provides durable storage, allowing you to replay events whenever necessary. Databases with Streaming Capabilities: - Redis: Redis is known for its in-memory storage, offering fast performance. Also, it has built-in publish/subscribe messaging features, enabling it to function as a real-time message broker for instant communication. #dataengineering #scalablesystems

If you find yourself wondering which Apache Kafka workloads to host on Express brokers 🚄in Amazon MSK, the answer is really simple: any mission-critical workload would benefit immensely from the best-in-class resilience that Express brokers provide. On top of that, if you have a workload that is storage intensive or one that has a variable traffic pattern, then you will be even better off using Express brokers for your clusters. How, you may ask? Let me explain First, Express brokers deliver fully managed storage that do not require you to provision or scale storage. By doing that, Express brokers allow you to scale your storage independently from compute, which can lead to excessive compute costs. Additionally, you can add/remove compute capacity quickly , which allows you to run your clusters at a higher average utilization than what you can with Standard brokers. With these two capabilities, you can actually save 💸💸 while also lowering your operational overhead 🥳🥳 AND improving resilience 💪🏽💪🏽