Zac Policzer

If you are working with Docker... I highly recommend looking at the Slim toolkit. https://lnkd.in/gX_Gb9Kf Why? Slim allows developers to inspect, optimize, and debug their containers using its xray, lint, build, debug, run, images, merge, registry, vulnerability (and other) commands. It simplifies and improves your developer experience building, customizing and using containers. It makes your containers better, smaller and more secure while providing advanced visibility and improved usability working with the original and minified containers. Don't change anything in your container image; minify it by up to 30x to make it secure, too! Optimizing images isn't the only thing it can do, though. It can also help you understand and author better container images. Keep doing what you are doing. No need to change anything. Use the base image you want. Use the package manager you want. Don't worry about hand-optimizing your Dockerfile. You shouldn't have to throw away your tools and your workflow to have small container images. Node.js application images: from ubuntu:14.04 - 432MB => 14MB (minified by 30.85X) from debian:jessie - 406MB => 25.1MB (minified by 16.21X) from node:alpine - 66.7MB => 34.7MB (minified by 1.92X) from node:distroless - 72.7MB => 39.7MB (minified by 1.83X) Go application images: from golang:latest - 700MB => 1.56MB (minified by 448.76X) from ubuntu:14.04 - 531MB => 1.87MB (minified by 284.10X) from golang:alpine - 258MB => 1.56MB (minified by 165.61X) from centos:7 - 615MB => 1.87MB (minified by 329.14X) Rust application images: from rust:1.31 - 2GB => 14MB (minified by 147.16X) Java application images: from ubuntu:14.04 - 743.6 MB => 100.3 MB

4

Open Policy Agent (OPA) and Rego are powerful tools for implementing policy-as-code in Kubernetes and other environments. OPA is an open-source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack. In Kubernetes, it's often used for admission control and authorization. Rego is the high-level declarative language used to specify policy as code in OPA. It allows you to express complex policies in a concise and readable format. Here's how OPA and Rego work together in Kubernetes: 1. Integration with Kubernetes: OPA can be deployed as a admission controller webhook in Kubernetes. This allows it to intercept and evaluate requests to the Kubernetes API server before they are persisted. 2. Writing Policies: Policies are written in Rego and typically stored in ConfigMaps or as files in the OPA container. 3. Policy Enforcement: When a request is made to the Kubernetes API, it's sent to OPA for evaluation. OPA applies the Rego policies and returns an allow/deny decision. Gatekeeper is an open-source project that provides a more Kubernetes-native way to use OPA (Open Policy Agent) for policy enforcement. It extends Kubernetes with Custom Resource Definitions (CRDs) for defining and managing policies using Rego. Reference for implementing gatekeeper for enforcing labels:

6

A new Jepsen report from Kyle Kingsbury looking at the Kafka-compatible Buf: https://lnkd.in/gijxaHzr It includes broader findings about the Kafka protocol and implementation too: "We also characterize four issues related to Kafka more generally, including the lack of authoritative documentation for transaction semantics, a deadlock in the official Java client, and write loss, aborted read, and torn transactions caused by the lack of message ordering constraints in the Kafka transaction protocol. These issues affect Kafka, Bufstream, and (presumably) other Kafka-compatible systems, and remain unresolved."

40

1 Comment

The final deep dive on change query support in the table formats is ready. This one covers Apache Paimon, focusing on understanding the change query support through the lens of its internals. https://lnkd.in/dFfM_8nA

28

See how simple it is to create chatbots with Microsoft Azure On Your Data and vector embeddings from an existing Elasticsearch index: https://lnkd.in/gBHZbkjq #ElasticSearchLabs #RAG #VectorDatabase

2

Introducing {quarto-tabby} - a new extension for Quarto documents that automatically creates tabsets for your code blocks making it quick to present code implementations in parallel. This is an early preview - more features coming soon! 📚 Documentation: https://lnkd.in/g4m9RQXb 💻 GitHub: https://lnkd.in/gRgPq4Gu #quarto #rstats #python

18

We need to move past the idea that knowing everything is the best; asking good questions is a much more critical and underrated skill that we need to embrace. I’ll demonstrate with a case study on how the exact problem to solve can dramatically shift based on answers to basic questions. But first, why does this matter? This is because good questions are the cornerstone to truly understanding the underlying requirements. So many issues would simply not exist if the right questions were asked at the right time - Many products with little to no adoption would not be built - Numerous designs would not be scrapped very late in the game - Countless incidents would be caught before they make it to production Let’s use my favorite leveling question shared previously https://lnkd.in/gxCHTWSu. "Imagine you join our team/company X and discover that our SLAs (Service Level Agreements) are consistently in the red. What steps would you take to address this issue?" Fundamental questions to ask: Q1: Are the customers actually unhappy? If there are no customer complaints, the problem may boil down to a simple SLA update. If there are complaints, changes to the system are likely necessary Q2: What specific SLAs are in red? How long is “consistent”? If multiple SLAs (latency, availability, consistency) have been red for 12 months, it's a deeper systemic problem. If a single SLA is red for 3 months, it’s a simpler problem. Q3: What is the SLA and what is the system performance? If the SLA and system performance don’t diverge that much, it can be solved with incremental steps. However, if the SLA is 99.99% and performance is 99%, then deeper issues must be addressed. To recap… Asking the right questions at the right time can entirely transform the approach and solution, saving a lot of time, effort and grief. It is a critical skill for senior engineers. I’ll leave you with this iconic clip from Margin Call… “Explain, as you might to a young child. Or a golden retriever” ~~~ ✍️ Do you agree? Share if you have examples where the right question changed project’s direction. ♻️ Repost if this is helpful.

30

2 Comments

It was hard, but we now have a primitive at Gadget that lets us do absolutely 0 downtime atomic changes to an application's structure or data under the hood, with 0 requests dropped. I call it "request catching" but it was inspired by this Braintree eng post I read long ago For stateless workloads like serverless functions, 0-downtime deploys are pretty easy. You stand up the new one next to the old one, use a load balancer to start sending requests to the new one, and slowly stop sending requests to the old one. But for stateful workloads, it's much harder. You can't just start a new Postgres and send some queries to the new one and some to the old one without lying to somebody about state. If you want to be consistent, there has to be one source of truth. Fancy DBs have quorums for this, but PG doesn't. If you want to be consistent, you're effectively forced to have some tiny-but-real window of downtime in between draining the old one and starting the new one. That said, there's nothing forcing you to return errors to clients while that is happening. If your infrastructure supports it, you can pause requests while this changeover is happening under the hood, and then resume them once it is over. Clients will notice a longer request that normal, but it will be served like normal and they'll be none the wiser! What's required to do this though is a deep deep integration between the load balancing infrastructure and the thing doing the database deploy under the hood. We've built this up using a distributed locking tool, where all requests start by taking out a shared lock. When we're ready to do the maintenance under the hood, the maintenance workflow takes out an exclusive lock, blocking all requests from proceeding. The workflow does the atomic change it needs to make knowing it has exclusive access which means nothing will change under its feet, and then releases the exclusive lock as fast as it can so those requests can proceed. As an optimization, we only check the lock on the request path if an app is marked as near a maintenance window. This is only possible because of Gadget's architecture where we bundle together the load balancer, the request processing layer, and the database access tier such that we can reliably intercept all requests and all database transactions and do this lock checking. Its these nasty, in-the-weeds details that make me so proud of Gadget -- we can do world-class engineering stuff under the hood on behalf of all our apps without them ever noticing. If you're interested in doing this on your stack some OSS tools have the primitives you need! We use nginx for our load balancer, Temporal for the reliable workflow execution and we're currently using postgres advisory locks for hte locking. I think the Braintree post has been lost to time after they got acquired, but @simonw has a great post on the subject here: https://lnkd.in/eZmiktGT

51

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

The Unsung Heroes of Tech Site Reliability Engineers (SREs) play a crucial role in today's tech landscape. They blend software engineering and IT operations to ensure systems are scalable, reliable, and efficient. But what does a typical day look like for an SRE? Morning starts with a review of system metrics and logs. SREs check for any anomalies or potential issues that might have occurred overnight. This proactive monitoring helps in identifying problems before they escalate. They use tools like Grafana and Prometheus to visualise data and set up alerts for critical thresholds. Next, they dive into incident management. If any issues are flagged, SREs work on troubleshooting and resolving them. This could involve debugging code, liaising with development teams, or even rolling back deployments. The goal is to restore service as quickly as possible while documenting the incident for future reference. Afternoons are often dedicated to improving system reliability. This includes automating repetitive tasks, refining deployment processes, and enhancing monitoring systems. SREs might also work on capacity planning, ensuring that the infrastructure can handle future growth. They collaborate closely with developers to implement best practices and optimise performance. A key part of the role is continuous learning and adaptation. SREs stay updated with the latest industry trends and tools. They attend training sessions, participate in webinars, and engage with the broader tech community to share knowledge and insights. Interested in the world of SRE? Comment below or connect with me on LinkedIn if you're looking to hire or explore new opportunities. Visit charles-simon.co.uk for more information. ✅ #SRE #TechJobs #ITRecruitment

1

This is a very interesting article on sparse vectors, which can help overcome limitations of vector search related to domain specific keywords. And of course, the code in the example is clear and clean! Sparse vectors have a few values and many zeros. Dense vectors have limited or no zeroes. But really, what we're talking about semantic search on terms where the context is not present or harder to infer. The author presents this as specialized keyword search such as medical terms or other domain specific terminology. "These scores reflect the significance of each term in the context of the document or query, guiding the model to allocate more weight to terms that are likely to be more meaningful for retrieval purposes." Highly recommend reading. Nice work Paul Iusztin. And a nod to Qdrant and their vector search technology that supports sparse vectors. https://lnkd.in/gcBWfTg8

3

The subject of support for incremental reads of mutable tables and generating CDC streams in the four table formats I'm covering is far more nuanced than I expected. What was going to be one post is going to have to be a 5 parter to keep it manageable: 1. Introduction to the concepts 2-5. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon

15

2 Comments

In this clip from Orb's Tractable podcast, WarpStream's Richard Artoul talks about how WarpStream is a drop-in replacement for Apache Kafka and its zero disk architecture means you never pay interzone networking fees. Check out the full episode.

1

Excited to see https://lnkd.in/gG7fp73Y go live this Kubecon! The brainchild of the brilliant Amine Hilaly, KRO leverages the strengths of the Kubernetes API and GitOps principles, allowing organizations to package and distribute deployment recipes for both applications and infrastructure. KRO simplifies the industry's patchwork of deployment tools with Kubernetes APIs and a server-side DAG execution engine. Excited to see this mature! #kubecon #kubernetes #eks #opensource

96

10 Comments

It seems that with C/C++ -> Rust, specifically, you could do this by comparing the output compiled code as one of your verifications of correctness for ML learning This is something that e.g. copilot doesn't do - but it'd be straightforward to do so when doing training The expectation would be that the compiled rust code should be equivalent to the compiled C code, i.e. the assembly should be semantically no different As the ML learns, it may end up doing some funky rust code simply to get that to match - but that's probably a good thing, as this is the part that would require some manual intervention & you'd at least be guaranteed equivalent behavior

1

Come hang out with me this Friday, May 3rd @ 1 PM EDT for a free NFJS (No Fluff Just Stuff) webinar: Feature Flagging at Scale: Considerations For Organizational Success Feature flags can be transformative for organizations, but they're not a one-size-fits-all solution. As teams grow and projects expand, unique challenges arise. In this session, we'll explore the critical considerations organizations must keep in mind to ensure feature flagging paves the way for sustained success, offering insights into how SPS Commerce, the world’s largest retail network, has matured its feature flag adoption. https://lnkd.in/g8FpTxW8 #NoFluff #FeatureFlags #DeveloperExperience

21

1 Comment

The future of app development is here. Redis Released is our one-day event for developers to connect with Redis experts, learn about our product roadmap, and hear from the world’s largest brands on how they’re building with our powerful real-time technology. Here’s what we’ll share: → Our Product Roadmap → The World of Redis → Faster GenAI Apps Next stops: London on September 12th and New York City on October 17th. See you there: https://redis.io/released/

40