Jacek Chmielewski

Yesterday at Devoxx Poland in Krakow, I gave an updated version of my talk, How to Query a Stream. Herewith the tl;dr: Kafka has become the de facto data substrate of modern systems. It provides all kinds of benefits, but at a cost; to wit, it's very difficult to ask questions about what's in a log. There is no one way to solve this problem, so I proposed several distinct options. 1️⃣ You can always Kafka Connect a topic into your favorite RDBMS (what is it and why is it Postgres?), then query the resulting table. This might not scale to huge data sizes, but the infrastructure is something you're more likely to have around already. 2️⃣ If you're a Java shop, you can use Kafka Streams. It's straightforward to make a topic into a table queryable by message key, accessible through the Interactive Queries API. And if it's analytics you want, you can build an arbitrary stream-processing topology that again produces a queryable table. 3️⃣ There are sundry streaming SQL solutions like Materialize, Delta Stream, Rising Wave, and ksqlDB. They provide similar levels of stream processing and queriable materialized views. 3️⃣.5️⃣ It's here that Flink enters the chat! It famously does not expose its state store to be queried, but it's typical for the last-mile serving-layer-ish query solutions we're discussing to act like a key/value store. This means the computation that normally takes place in the Optimize/Plan/Execute flow of a regular database needs to happen somewhere else, and Flink often fills that role. 4️⃣ Cloud data warehouses. A big place to put all your data for the purpose of analytics. Like data warehouses of yore, these contain explicitly structured data. Oddly, the logos of all the common solutions are blue. I heard you like blue. 5️⃣ Data lakes. A big place to put all your data for the purpose of analytics, but for when you want to pretend structure doesn't matter for a little while. Open table formats like Iceberg are emerging because structure never doesn't matter. 4️⃣ and 5️⃣ are converging (in principle, it not in implementation) into the data lakehouse—a reflection of the decades-long struggle with the schema/no-schema question. 6️⃣ Real-time analytics. A specialized category of analytics infrastructure that ingests directly from Kafka and makes results available at extremely low latency and extremely high concurrency. A more and more necessary part of the stack as analytics escape the dashboard and become part of the end-user experience. We are collectively in a period of exploration of untamed territory between Kafka at the base of the new stack and the differentiated application code at the top of the stack. That code at the top should be the focus of most of our efforts as developers. The application stack was well-defined in the client-server/monolith era, but in the new streaming era, so much is up for grabs and yet to be defined. All cliches aside, it's a very exciting time to be a developer.

467

16 komentarzy

Developer Experience is still a problem! I listened to this podcast https://lnkd.in/eZPvJnGE on how LastPass tries to make local experience much easier in the area of microservices. They went from the usual suspect of Docker compose but it requires you to build Docker images before being able to run things. Then they went to local kubernetes but it adds another layer and it doesn't help with the extra Docker build step. They decided to use nix which is Docker without Docker, universal package manager. This is something I looked at when working on codefly agents. One goal of codefly is to provide great Developer Experience, a developer starts any service, it will run any dependent services, do all the networking, and even provide hot-reloading for languages that don't have it (golang for example). I found nix to be too complicated so I am relying on Docker and spent quite some time getting that right: you really don't want to start/stop containers all the time and it's actually not that trivial to start and stop processes inside a Docker container. But the pain is worth it: - dependency is minimal, Docker is OK, nix is another story - performance hit is minimal since we try very hard to keep the containers around And agent can also use local stack if available. The philosophy of codefly is that your code needs some "co-pilot" for everything from setup, package maintenance, to networking, Docker build, kubernetes manifests. We do all the annoying stuff so you can just focus on writing code, writing tests that can depend on your full stack without any change of code, or mocking -- just that is a game changer and that's it. We take care of all the painful stuff so you can do the fun stuff!

1

Instant performance is no longer a myth! In our upcoming webinar, we'll look at Five Modern Ways to Beat Latency: - Speculation Rules - 103 Early Hints - Volatile Content Acceleration - 0-RTT - Distributed Compute Cu on October 16th? Robin Marx and Tim Vereecke #webperf #instant #corewebvitals

60

This is my new favorite #toiletreading (the book you read at... ehm... certain place 😁) I got from guys from Particular Software (authors of NServiceBus) at #NDCOslo. Rather short book, but full of great examples of why distributed systems are hard and why some problems, especially reliable communication, remain the same for centuries. If you would like to buy it, here is the link for Amazon: https://lnkd.in/dXtwdS_x

5

1 komentarz

About Malte Ubl (Vercel's CTO) talk on "Framework Defined Infrastructure (FDI) – an Evolution of Infrastructure as Code (IaC)". Some thoughts: - FDI seems to be a specialized approach than a direct evolution of IaC. - The core idea behind FDI - allowing devs to work within familiar framework contexts - highlights the growing demand for more abstracted infrastructure interfaces. - FDI and similar concepts demonstrate the need for IDPs that can sit on top of IaC, providing a more developer-friendly experience without replacing the underlying IaC principles. While I'm not convinced FDI is the next step after IaC, it reflects the ongoing efforts to make infrastructure management more accessible to developers and obvious demand for IDP experience. 🔗ref https://lnkd.in/gU-9wT5J #InfrastructureAsCode #DeveloperExperience #FDI #DevOps #IDP

5

The GoodData hackathon is in progress. Few notes: - Inviting external teams (Keboola, Operator ICT/Golemio) was a really good idea - You always find incredible bugs in production during hackathon. External teams are even better at finding such bugs - JetBrains Intellij is now really broken, esp. when working with Python

26

5 komentarzy

🔭 OpenTelemetry Collector: The Architecture Review I have just published the second article in my OpenTelemetry series about design, architecture and interesting implementation spots in the OTel Collector including: - 🔗 The Signal Processing Pipeline Architecture -📡 OTel Receivers. Prometheus-style Scrapers - ⚙️ OTel Processors. The Memory Limiter & Batch Processor. Multi-tenant Signal Processing - 🚚 OTel Exporters. The Exporting Pipeline & Queues. The implementation of persistent queues - 🔭 How observability is done in the OTel Collector itself. Logging, metrics, and traces - 🔌 OTel Extensions Design. Authentication & ZPages -👷Custom Collectors & OTel Collector Builder - 🚧 Feature Gates Design & The Feature Release & Deprecation Process Enjoy the article & let me know how do you like it 🙌 #observability #opentelemetry #golang #distributedsystems

17

3 komentarze

The benefits of AI code assistants are still being evaluated, but here at Deutsche Telekom we are confident that our AI strategy to continually assess COTS products, as well as developing our own products and data foundation is the right approach to yield benefits in the future. Have a look at the post below from one of our leading developers on the topic It’s exciting to think about the capabilities of the product in only a few years from now; perhaps covering all aspects of the SDLC and integration to user stories, more context in relation to existing code repos throughout the enterprise (particularly useful for very large enterprises), models optimized for dialects of the languages (GoogleSQL vs PostgreSQL etc)….to think of a few examples Regardless, the developer experience will be greatly enhanced in the next few years and at Deutsche Telekom we’ve got a front row seat! 👏

16

1 komentarz

🚀 🚀 🚀 Attention all AI enthusiasts and data wizards! 🧙‍♂️📊 Our Vendavo Engineering Blogpost is in full swing as we head into the weekend. Our very own Peter Aláč had the chance to attend the 2024 Machine Learning (ML) Prague Conference recently. In the below blogpost, Peter shares his insights, as well as the most relevant and beneficial sessions from his point-of-view. 🎉🎉🎉 From mind-blowing insights to cutting-edge tech, this conference was a true game-changer. Conferences like this always provide a great treasure trove of ideas to revolutionize our AI and ML efforts! 💡💻 Finally, Peter touched upon the networking opportunities that existed. These events are a great chance to meet some of the best and brightest minds in the industry, as well as, connect with like-minded data aficionados. It was a networking extravaganza! 🤝🌟 Enough from me. Click the link below to dive into the journey and discover the secrets that Peter uncovered at ML Prague 2024. Trust me, you'll up your machine learning game, and be the life of the next networking party you at! 🚀 #MLPrague2024 #AI #MachineLearning #DataScience #Vendavo #Innovation #TechConference #Networking #GameChangers https://lnkd.in/gKwj82ZC

10

So I became fascinated by C++ coroutines late last week, and have spent a good portion of the weekend figuring out how they work. The TL;DR of my discoveries so far is that C++20 coroutines: 1) follow the best and the worst of C++ practices, and 2) are a really clever way to decouple stackless functions from their execution model. For (1), well, it's C++. No crisp & clean syntax, lots of weird compilation errors, and even weirder runtime errors. I hated them with passion for the first several hours. But then they grew on me, because the flip side is also true. The flip side is backwards compatibility and "only pay for what you use". And introducing a totally new concept — the stackless execution model, effectively, [proto-] green threads — into a mature and complex language is one hell of a pain! For (2), I do believe it's a genius solution for the language to separate stackless functions from their execution model. And C++20 did exactly that. There is literally no event loop in C++20! The claim (as I understand it) is that a tiny microcontroller may want to use C++20 coroutines, and the authors of the Standard are very explicit about not constraining possible runtimes. All that C++20 coroutines do is introduce stackless functions into the language as first-class citizens. And yes, stackless immediately implies resumable. So it is one step about the most-clever-possible syntactic sugar over `setjmp()`. Full one step. But not a millimeter more than this one step. Whether you "execute" your coroutines in a dedicated single-threaded event loop, or a thread-pool-based event loop, or by individual steps via some timer event or even a network packet — that is entirely your call. The stackless functions are there, `co_await` (as well as `co_yield` and `co_return`) is there, no callback hell is needed, and all the RAII and scoped initialization/ownership model is fully intact. Now, I'm still elbows deep in my experiments, but here's the first big LinkedIn-worthy question. Why don't we write our C++20 [unit] tests as corourines? Say, you want to limit some test to five seconds. And fail it as "timed out" within these five seconds. Before coroutines, doing this without instrumenting the very code of the test involved `fork()`-ing the process, effectively relying on the operating system's primitives to ensure proper resource deallocation. With C++20 coroutines, as long as the slow test is slow on non-blocking I/O (i.e. not in an infinite loop, but on something that `co_await`-s stuff), it is possible to time-box each individual test to some five seconds very much organically. Because a coroutine frame, unlike a stack frame, can be `.destroy()`-ed safely and cleanly. So, are we doing it yet? If no, why not? If yes, what would be some next-gen `gtest` to look into? Also, I'll be writing some code and blog posts on C++20 coroutines soon. If you are reading this text and are an expert, DM me pls — I'd love to pick your brain!

19

4 komentarze

Interesting read from Anatoly Zelenin about why backing up your Kafka data is important! https://lnkd.in/eZVbX_Qq One of the main causes of data loss is are due to human errors. Topic misconfigurations, release of a bug which corrupts your data, deletion of topics, ... Although it's a blogpost from back in 2019, it's even more relevant these days as more and more companies are using Kafka as the backbone of your IT infrastructure. So back up your events, you'll thank yourselves later! #kafka #backup

8

1 komentarz

Creating images with #Opensource #AI is super cool, but it can be dangerous if not done right. 🖼️🤖 A recent #article (https://lnkd.in/dDRTPpYc ) on @vpnMentor highlights the risks associated with using Opensource AI tools like #ComfyUI. A malicious custom node was introduced into the system, potentially compromising user data and security. This incident serves as a crucial reminder to always use trusted sources and keep our security protocols up to date. Stay safe and enjoy the creative power of AI responsibly! #AI #CyberSecurity #ImageCreation #TechSafety #ArtificialIntelligence #DataProtection #AIArt #DigitalSecurity

6

Experimenting with visual topology builder support in my #topoSphere SDK led me to an interesting Proof of Concept for #Containerlab, a versatile network emulation tool made by Roman Dodin of Nokia. This PoC features on-the-fly two-way topology format conversion between topoSphere and Containerlab. Two core design considerations that make this possible: – topoSphere implements node drag&drop handler and edge drawing handler, allowing developers to pass optional callback functions through the config. If specified, these functions are executed before the node or edge creation and can return the additional properties or cancel the creation. – topoSphere emits custom events on node and edge creation. The combination of these features should allow the developer to build any flexible workflows around the visual builder. Any professional developers out there, how would you approach implementing a feature like this? Stay tuned for more updates and feel free to share your thoughts and suggestions.

98

11 komentarzy

Check out this blog post from Inside Open Source where we discuss how we are rebuilding the cloud stack, taking a first principles approach, to eliminate common headaches such as exploding cloud bills, cold starts, slow autoscale or sluggish scale to zero. Thanks David Meiborg and Ari Joury, PhD for the opportunity!

22

My KubeConNA presentation with Dr. Stefan Schimanski is out! Thanks to all, who attended. Normally deep-dive sessions are not so popular, so we were a bit skeptical: "Maybe we will get 30-40 people" we said. So how surprised we were when we got around 400 registered and ~300 in the room :D It's always a challenge to fit a deep dive into 35 minutes slot as you end up spending half-time setting the baseline, and the rest of the time "deep diving". But we are happy with the result! Amazing questions and encouraging comments. There is a need for Kubernetes-like control plane technology out there! And now we have 2: generic control plane and kcp! We heard the feedback: 1. Make it easier to consume, easier to deploy 2. We need to refine (for the third time) the compute story with generic control planes (maybe it's time for some diagrams?). 3. Tenancy management is complicated and unclear—point taken! Way to provision your own tenancy forests instead of a single tree! 4. Better documentation. (Anybody wants to help us? please get in touch!) 5. And finally - you liked the "mount concept" - it is a clear signal we need to move out of the experimental phase to something part of the core! All and all - Thanks for being part of this! Every talk like this requires an enormous amount of time and effort and seeing all of you there - motivates us more. Thanks #CastAI #Upbound #Kubermatics #CNCF (and a few others which I not gonna tag because you want to stay stealth :) ) for the support! Democratize control planes! https://lnkd.in/dRQADcCs https://lnkd.in/d2W7PnVN

43

What if? Yesterday I convinced again that often “limits” in documentation are not in reality. Like it’s written in Elon Musk biography, limits are important, but they need to go in line with physics law, simple logic and luck. If you have not read it, please do 😊 I was surprised by my 5-years old laptop Lenovo T480. It has 32GB RAM all the time from 2019 and for some tasks it’s not enough. I read all specifications for laptop and Max RAM size was mentioned 32GB. Some time ago I bought for my work laptop two RAM modules with 32GB each (64GB in total) and used it around the year. Then I changed work and these two RAM modules moved to “spare parts” box. Yesterday I was working on my old laptop and asked myself “What if laptop supports 32GB modules? What is the maximum risk? Laptop will not start with 32GB modules and I can put 16GB modules back. Let’s try”. I quickly replaced the modules and it worked! I asked myself – why it was not mentioned in specification documents for laptop and CPU? Looks like in 2019, when laptop was produced there were no 32GB RAM modules for laptops on the marked and Lenovo and Intel did not check and as result did not add it into specification. This is remined me how Elon Musk asked his engineers regarding space rocket production standards and limitations – “Who created the limit and when?” Some limits for space rocket constructions and not only were developed decades ago when some technologies were not developed and engineers need to review it. It’s always good ask yourself “What if?” if your idea does not break physics law and simple logic. In most cases you’ll be lucky 😊

5

A bit from my personal #selfhosting projects, making it high-available as KISS (keep it simple and stupid) as possible: https://lnkd.in/d77ntydy #openbsd #dns #failover

1

Excited to host a session at NG Poland Conf this week! 🎙️ In this session we are going to review the Core Web Vitals and make use of some directives and techniques that have been available in latest #Angular versions to improve the Web Vitals of a sample application. If you're interested in the topic, here is the link to the sample app's repository: https://lnkd.in/dKy3Z5Sc

26