Gwen (Chen) Shapira

Application developers want to use more and more #LLMs -- platform engineers need to make AI resources available, strip out authentication friction and avoid shadow IT, or general chaos. Ideally, they point developers at a single "AI Gateway" sitting in front of those disparate #onprem or #cloud models. Bloomberg needs this -- and wanted to avoid vendor lock-in. Its platform engineers are teaming up with Tetrate to build and open source a new #AI Gateway built around the #Envoy project. We have more 👉 https://lnkd.in/e8ftZHW7 h/t Yuzhui Liu, Dan Sun Chaim Haas

14

1 Comment

Join me next week for a discussion about table partitioning at the inaugural 🐘 Postgres Meetup for All 🐘 virtual meetup. We'll talk about partitioning basics, the why and how, and even a few things that recent versions of Postgres have improved. https://buff.ly/4eRUJUU The goal for "Postgres Meetup for All" is to provide not only a variety of speakers and topics, but time for discussion and interaction. I'm planning on a round of introductions from attendees, a short presentation (20-30 minutes), and then discussion time. If you're thinking about table partitioning, working through issues, or even have some tips of your own to share, join us for this discussion.

17

Adam's post is good (and I like this categorization in general), but there are a couple of things that I don't agree with. First, I'd say time-to-recovery is a strength of CfD rather than CfM. Physalia (from our paper "Millions of Tiny Databases" https://lnkd.in/gepZPA2Y) is the CfD system that provides the metadata for EBS (https://lnkd.in/g7R3MgYk) a vastly bigger CfM system. We chose CfD for Physalia specifically to optimize recovery time, because it needs to recover before the larger system can recover. Second, I think the post is making an assumption that CfM systems are f+1 on the dataplane, use the minimal number of replicas, and don't use quorums. Aurora is an instructive counter example (https://lnkd.in/gNrNTPp7).

117

2 Comments

StarTree, built on the open source #ApachePinot database, just announced several updates aimed at enhancing real-time analytics and observability. Originating from LinkedIn, Apache Pinot scales impressively and has won over major clients like Uber, DoorDash, and Walmart. StarTree's managed offering prioritizes real-time and external-facing analytics, but has expanded to include internal use, focusing on query capabilities and #storage for observability. The announcement highlights StarTree's integration with popular tools and frameworks for observability, the addition of vector index support for #AI use cases, and the introduction of StarTree #ThirdEye for sophisticated anomaly detection and root-cause analysis. Furthermore, StarTree has revamped its data ingestion capabilities with a new write API and has shifted from a trial model to a free-forever plan. #Rundown

16

1 Comment

VPC with zone subnets, alb, nat and igw https://lnkd.in/gZxkgdBx

5

📢 Big News in the Kafka Ecosystem! 📢 Confluent has acquired WarpStream for $220M, just 13 months after its launch. This isn’t only a business milestone—it’s a huge technical advancement for Kafka users and cloud-native architecture. Here’s what makes WarpStream’s approach groundbreaking and how Registfy can help businesses leverage it on AWS: 🔹 Diskless Kafka Architecture with S3 WarpStream redefined the Kafka-API with a cloud-native, diskless architecture using S3 as storage. By eliminating disks, they achieved massive cost savings and operational simplicity, resonating deeply in the DevOps and cloud solutions world. 🔹 Incredible Cost Savings & Operational Efficiency WarpStream’s design is 10x cheaper than unoptimized Kafka clusters and 4x cheaper than optimized deployments. For high-throughput use cases, this can mean savings in the millions:     Unoptimized Kafka: ~$5.25M     Optimized Kafka: ~$2M     WarpStream: ~$500k 🔹 Latency Trade-Offs for High-Volume Workloads With a latency of 400ms (p99 writes) and 1 second end-to-end, WarpStream’s approach works best for high-volume, latency-insensitive workloads. Many customers are willing to trade latency for such remarkable cost benefits. 💡 Key Insight from WarpStream By focusing on high-volume, latency-insensitive workloads, WarpStream delivers incredible cost-effectiveness—revolutionizing how businesses might consider their Kafka deployments. 🚀 How Registfy Can Help As an AWS Advanced Partner specialized in DevOps and cloud solutions, Registfy is uniquely positioned to help businesses implement cost-efficient, scalable Kafka architectures on AWS. With WarpStream’s acquisition by Confluent, this opens new opportunities for customers seeking optimized cloud-native solutions. Let’s discuss how to make your Kafka workloads more affordable and scalable. #Kafka #CloudNative #CostEfficiency #Confluent #WarpStream #AWS #Registfy #DevOps #TechInnovation

6

Check out Postgres expert, Ahsan Hadi's new blog post out today -- a follow up to last week's post -- entitled, "PostgreSQL 17 - A Major Step Forward in Performance, Logical Replication and More." In this blog, Ahsan describes a couple of new performance features that you'll find in Postgres 17 as well as another important logical replication feature that he didn't cover in his last blog in this series. Learn more and read the blog today! https://hubs.la/Q02T3X6d0 #pgedge #postgresql #postgresqlprofessionals #logicalreplication

21

Developers can now store, index, and query vectors in PostgreSQL using pgvector, eliminating the need for a dedicated vector database. This new blog from Yugabyte expert Brett Hoyer deep dives into pgvector, and explains how to store text embeddings as vectors, execute similarity search, choose between indexes, and scale with distributed SQL.💡 Find out more!⬇️ https://hubs.la/Q02CNjWd0 #YugabyteDB #pgvector #vector #AI #Postgresql #cloudnative #distributedsql

10

Cloud cost optimization needs to shift left, Listen to why/how with Flex Compute, Listen to Khursheed Hassan on #cloudnativefm: https://lnkd.in/e42HB_A2 where cost prevention is the leading strategy, and cloud cost optimization tools are only deployed in a downstream process to catch a tiny number of usage anomalies that have slipped through the tight net of preventive checks Today there are 3 ways to provision capacity on Cloud √ On-Demand Instances: While they provide flexibility, they are often associated with cost inefficiencies and unpredictable expenses since the billing is based on usage per hour or minute, without any long-term commitments. This can cause costs to spiral out of control if resources aren't managed properly. √ Spot Instances: Though these offer cost savings, they come with their own set of challenges. They are less predictable, require complex orchestration, and introduce additional complexity to cloud infrastructure. There can also be latency issues, and instances may be reclaimed by the cloud provider without prior notification, leading to disruptions. √ Capacity Reservations: This option involves long-term commitments, typically ranging from one to three years. While it guarantees resource availability, there is a risk of underutilization. If resources are not actively used, organizations may still incur high costs, making this solution potentially expensive when workloads are not running at capacity." CloudiDR is building a solution around flex compute that effectively address non-steady state workloads, which inherently rise and fall. They’ve launched a product suite of tools around flex compute as Cyber Recovery Compute,' which can also be used for Disaster Recovery purposes. and  'Scheduled Capacity Compute.' their thesis is that and don’t want to replicate what cloud providers are already offering. Instead, focus on identifying unaddressed gaps in the market and creating innovative solutions that deliver the best value for their customers. Their solution refer to as Cyber Recovery Compute (or Disaster Recovery Compute), allows customers to avoid paying full reservation prices and sidesteps the unreliability of on-demand options. Instead, they can access guaranteed capacity from Cloud IDR at 75% lower costs. Their solution refer to as Schedule Capacity Compute, is specifically designed for workloads such as CI/CD pipelines or events. For instance, if a concert broadcast is scheduled or there are retail events featuring promotional coupons, we anticipate a significant increase in compute demand. Customers can schedule their compute resources with Cloud IDR in advance, ranging from as little as five minutes to several weeks ahead of time. They will then transfer the compute resources into their account at a minimum of 30% lower cost compared to on-demand pricing. Listen to and watch the full episode: https://lnkd.in/e42HB_A2 #cloud #hyperscalers #shiftleft #cloudoptimization

19

We recently started a new LinkedIn Live series for everything Apache Hudi and Open Lakehouses 🎉 In the very 1st episode, Dipankar went over the fundamentals of Apache Hudi and showed hands-on examples of running transactions with compute engines like Apache Spark. If you missed the live, here's the recording: https://lnkd.in/dm-24JzC Check it out and share with the Hudi community! #dataengineering #softwareengineering #lakehouse

15

💡 The Redshift team at AWS recently published a study analyzing half a billion queries on 32 million tables over a 3-month period. The findings? 67% of queries scanned less than 1 GB of data, and 99.8% of scanned less than 10TB. This reveals a reality check—most data workloads are far smaller than we often assume. So why overcomplicate things with massive infrastructure? Embrace the simple joys of small data

234

11 Comments

To leverage preemption, follow these best practices: ✅Define PriorityClasses for your applications based on importance. ✅Assign PriorityClasses to pods appropriately. ✅Monitor cluster load and pod evictions. ✅Adjust PriorityClasses as workload priorities change. By mastering preemptible pods in Kubernetes, you can keep your critical services running smoothly and optimize overall resource utilization. Give it a try on your next cluster upgrade! https://hubs.la/Q02LrySX0 #kubernetes #k8s #sre #devops #devtools

15

What's the best way to handle an online schema migration in MySQL? PlanetScale engineer Shlomi Noach provides an overview of available options and weighs the pros and cons in our latest blog post. https://lnkd.in/gdR3CYvM

17

Optimize your MongoDB performance with MinervaDB's best practices, including choosing the right storage engine, disk I/O, memory tuning, replication, and sharding. https://zurl.co/VsOI #MongoDB #NoSQL #DocumentStore #DBA

7

"To ingest the incremental data from Kafka to S3, we considered three excellent data lake and ingestion solutions: Apache Hudi, Apache Iceberg, and Delta Lake. In the end we chose Hudi for its excellent performance with our update-heavy workload and its open-source nature & native integration with Debezium CDC messages." Excellent blog by Notion's data platform team on how they built their data lake (house) to meet the surge in 10x data growth due to user & content growth to cater to AI, Search, and other product use cases. 🔗 Blog: https://lnkd.in/dEYxAZDQ #dataengineering #softwareengineering

62

1 Comment

📢 Real-Time Data Replication with Debezium Maintaining data integrity across multiple systems is challenging, especially with a distributed setup like Kafka. Some adjustments are necessary to get it right. Check out the details in our latest blog post! 🔗 https://lnkd.in/eMi377dj #Debezium #DataStreaming #Kafka #RealTimeData

16