Jon Haddad’s Post

Kafka has been there for quite some time. Everybody and their uncle uses it for data streaming and processing. You know it can perform, and we know it. But how do you get the most bang for your buck out of it? We rigorously tested Kafka performance, comparing different environments to find the most cost-effective setup in AWS or GCP. We ran dozens of tests to find which compute generation, CPU, and JVM would yield the best price-performance ratio for Kafka. Specifically, we measured how many millions of rows we can ingest into the Kafka broker per one cent. Read more in our blog (spoiler: ARM rocks): https://lnkd.in/eJ7aRpbw

https://lnkd.in/dHTYqJ6T [ main site: https://lnkd.in/d3mcNX-C ] << ...WarpStream is an Apache Kafka® compatible data streaming platform built directly on top of object storage: no inter-AZ bandwidth costs, no disks to manage, and infinitely scalable, all within your VPC... >>

"Our cloud database stores billions of files in object storage. With petabytes of data being queried every day, we started bumping into our cloud storage providers' rate-limits, resulting in decreased reliability & performance. We had large memcached clusters in place to absorb & deamplify reads to object storage - but these could hold at most a few hours’ worth of data, and constantly churned due to the excessive volume of data passing through. The conclusion we came to was: we needed much larger caches, ideally without inflating our cloud costs and adding operational complexity. I'll show how we managed to increase our cache size by 45x and reduce our costs by using a little-known feature of memcached called "extstore". Extstore enables offloading of objects to SSDs which can't fit into memory. In this talk I’ll be covering how we use it, how to monitor it, why we chose it, and other considerations. I'll also cover how we use ephemeral storage provided by public cloud vendors in the form of physically-attached SSDs with incredibly high throughput, low latency, and best of all - low cost! This talk is also a story of how products evolve, and how we as a team are buying time in the short term to keep up our reliability while we evolve our storage design in the medium-long term." https://lnkd.in/eT8qbGcm

Just published part two to this series, explore the magic of serverless computing as we link Lambda functions to the database!

What is #ApacheFlink? How does it complement #ApacheKafka? Why is it different from #KafkaStreams? How does the #cloud help with #streamprocessing? => I discuss all these questions in this lightboard video in under ten minutes... https://lnkd.in/dsihChH2

What is #ApacheFlink? How does it complement #ApacheKafka? Why is it different from #KafkaStreams? How does the #cloud help with #streamprocessing? => Watch Kai Waehner discuss about it in under 10 minutes in this lightboard video https://lnkd.in/gEP39AZt

𝐃𝐨𝐧’𝐭 𝐋𝐞𝐭 𝐘𝐨𝐮𝐫 𝐄𝐊𝐒 𝐃𝐨𝐬𝐚 𝐁𝐮𝐫𝐧! 🍛⁣ ⁣ Running outdated EKS clusters can scorch your AWS budget faster than a dosa left on the tawa. AWS charges extra for older versions—up to $𝟎.𝟔𝟎/𝐡𝐨𝐮𝐫 𝐩𝐞𝐫 𝐜𝐥𝐮𝐬𝐭𝐞𝐫 if you delay upgrades. Avoiding these hidden costs is as easy as flipping your cluster on time. Want to know how to avoid paying extra for “burnt” clusters? ⁣ #awseks #costoptimisation #kubernetes #clusterupgrades ⁣ Check out the full story here: EKS Upgrades: 

What is #ApacheFlink? How does it complement #ApacheKafka? Why is it different from #KafkaStreams? How does the #cloud help with #streamprocessing? => I discuss all these questions in this lightboard video in under ten minutes... https://lnkd.in/dsihChH2

NATS JetStream is not just streaming but also a proper KV data store with many features beyond just Put, Get and Delete, and can hold it's own when compared to the reference that is Redis in many use cases. Another great video by Jeremy Saenz going through all the KV features currently available (and some of the upcoming 2.11 features) in NATS.

With the goal of reducing latency, I ordered an AWS Link encoder and will be testing ingest into AWS MediaConnect via both Zixi from this Link encoder and RTP with FEC from FFmpeg (./ffmpeg -re -i -c copy -map 0 -f rtp_mpegts -fec prompeg=l=5:d=20 rtp://:5000). I would love to get any feedback from people who have run similar tests with any configuration recommendations. Which one will have lower latency?