Ashish Dubey

🛠️ Build powerful applications with the integration of #AzureCosmosDB NoSQL and #OpenAI! Sandeep Nair demonstrates how to leverage vector embeddings for seamless multi-modal data operations. Watch now: https://lnkd.in/ezx6s8yu

2

Amazon RDS Multi-AZ deployment with two readable standbys supports 6 additional AWS Regions Three-node Multi-AZ DB Cluster is now available in Asia Pacific (Hyderabad), Asia Pacific (Melbourne), Europe (Spain), Europe (Zurich), Middle East (UAE), and Israel (Tel Aviv) Regions. This deployment option supports minor version upgrades and system maintenance updates with typically less than one second of downtime when using Amazon RDS Proxy or any one of the open-source AWS Advanced JDBC Driver, PgBouncer, or ProxySQL. https://lnkd.in/g9D9RGVB #Amazon #RDS #PostgreSQL #MySQL

90

1 Comment

⏰ Only one week to go! Get ready for the Postgres Conference, Julian Fischer will be spotlighting Klutch, the open-source solution for managing data services, like a8s PostgreSQL, across many Kubernetes clusters. 🐘🚀 In this demo, you’ll get to see Klutch spin up a new PostgreSQL database remotely in seconds – Don’t miss out! 📅 Date - Nov. 7th at 11:40 PST 📍 Room - Ops: 421 Learn more about our a8s PostgreSQL solution here: https://lnkd.in/ecmqt5ry #PostgresConf #OpenSource #Klutch #PostgreSQL #anynines

1

Take that Azure Cosmos DB powered app to prod with the help from Sandeep and Saranya! Watch the three part video series: "Mastering #AzureCosmosDB: A Comprehensive Guide from Prototype to Production." https://lnkd.in/eEMt5TFG

1

Transform Your Data Operations with Confluent! At Xstream Minds, we understand the challenges of managing data across diverse environments. That's why we're proud to partner with Confluent to deliver a seamless, flexible, and scalable data streaming solution for any infrastructure. Public Cloud & Multi-Cloud : Leverage the power of fully managed services with Confluent Cloud to scale effortlessly across providers like AWS, Azure, and Google Cloud. Private Cloud & On-Prem : Ensure compliance and security by deploying Confluent Cloud directly within your on-premises or private cloud environment. Hybrid Cloud : Create a persistent bridge between your on-premises data center and the cloud, unlocking the full potential of your hybrid infrastructure. No matter where your data lives, Confluent provides true development flexibility to span all your environments and drive innovation faster than ever before! Let’s take your data streaming capabilities to the next level. Ready to explore how? Connect with us today! #DataStreaming #ConfluentCloud #HybridCloud #CloudComputing #DigitalTransformation

5

𝗦𝗲𝗺𝗶-𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗱𝗮𝘁𝗮 𝗱𝗼𝗲𝘀𝗻'𝘁 𝗵𝗮𝘃𝗲 𝘁𝗼 𝗯𝗲 𝘁𝗼𝗼 𝗵𝗮𝗿𝗱   Working with varied data formats that have complex or unclear structures is often challenging. Today's data ecosystem deals with more than just traditional rows and columns, with modern databases needing to process complex formats like JSON, XML, and Parquet.   Semi-structured data provides the flexibility to work with dynamic data without requiring a rigid schema. JSON has become popular because it's easy to read, lightweight, and works well for sharing data between web apps, APIs, and databases.   Snowflake has features that make working with JSON easier and faster, with less hassle and more insights.    To help with this, here's a 7,000-word guide on how to query JSON data in Snowflake. ⤵️   Comment what you'd like to learn about MongoDB and JSON data next, and we'll publish it. 😁 #dataingestion #blog #guide #snowflake #query #json #data #datazip #database

14

With the introduction of BigQuery’s physical storage billing model, users asked for greater understanding to what influenced the compression ratios. In this blog post, we run some experiments to illustrate just that. DO YOU WANT TO KNOW MORE #googlecloud #bigquery

22

1 Comment

In-memory data processing, similar to #Hadoop and #Spark in the past, provides discontinuity in terms of cost, performance, and flexibility. Our friends at MotherDuck compiled a list of key workflows that leverage DuckDB to revolutionize analytics. We are committed to bringing this innovation to the Data Product landscape through the GoodData Unified Analytics (BI/AI/ML) platform: 🌐 Zero-Copy 🚀 Lightweight Compute ⚡ Pipeline Processing 🖥️ Interactive Data Apps 🔒 Enterprise Security

56

1 Comment

How can we not be excited with #AI, #ApacheFlink and cost-saving news all in one place? Live from #KafkaSummit Bangalore, we've announced: ➡ AI Model Inference to incorporate real-time AI/ML into your streaming data pipelines ➡ Confluent Platform for Apache Flink to enable stream processing in private clouds and on-prem environments with streamlined support ➡ New auto-scaling Freight clusters for up to 90% savings at GBps+ scale Read our press release for all the details! https://lnkd.in/g6p3TWyc

68

In 2013, Jay Kreps introduced the concept of the log as a fundamental abstraction for real-time data streaming, paving the way for platforms like #ApacheKafka. Over the past decade, the data streaming landscape has undergone a remarkable transformation, particularly with the rise of cloud-native environments and the integration of scalable object storage solutions like Amazon S3. This evolution has given rise to platforms like #ApachePulsar, each offering innovative approaches to log storage and data streaming. In our latest blog post, we explore the evolution of log storage implementations in data streaming and categorize this architectural journey into three distinct stages: 💡 Leader-Based Architecture Leader-based architectures form the foundation of most data streaming technologies used today. Traditional message queues rely on single-node brokers with primary-backup setups for high availability (HA). Kafka was the first distributed implementation, but it remains leader-based, where a designated leader manages replication. Pulsar advanced this by decoupling compute and storage, but it still follows the leader-based model. This approach works exceptionally well for scenarios requiring ultra-low latency (e.g., milliseconds). For such use cases, sending data through a leader remains unavoidable. 💡 Leaderless Architecture The emergence of S3 as a primary storage layer has triggered a shift toward leaderless architectures. Instead of treating S3 as just a tiered storage solution, it now serves as the main storage layer. In this model, brokers become leaderless, meaning any broker can handle reads and writes. Platforms like WarpStream, Confluent Freight Cluster, and StreamNative’s Ursa Engine are adopting this approach, reflecting the growing trend toward distributed and flexible architectures. 💡 Brokerless / Headless Architecture The next frontier is headless (or brokerless) architectures, where metadata and data are entirely decoupled: Data resides in S3, while metadata is managed by a separate metadata or catalog service. With a standard specification in place, operations no longer need to flow through brokers. Vendors can simply adhere to the standard and build their solutions accordingly. This approach can enable seamless scalability and interoperability. Our Ursa Engine is heading toward this exciting future. Explore the full blog post to dive deeper into these trends and the journey from leader-based to brokerless architectures: https://lnkd.in/gCcmv3rU You can also watch our keynote presentation at Data Streaming Summit: https://lnkd.in/g6a3wHrC

61

Adam's post is good (and I like this categorization in general), but there are a couple of things that I don't agree with. First, I'd say time-to-recovery is a strength of CfD rather than CfM. Physalia (from our paper "Millions of Tiny Databases" https://lnkd.in/gepZPA2Y) is the CfD system that provides the metadata for EBS (https://lnkd.in/g7R3MgYk) a vastly bigger CfM system. We chose CfD for Physalia specifically to optimize recovery time, because it needs to recover before the larger system can recover. Second, I think the post is making an assumption that CfM systems are f+1 on the dataplane, use the minimal number of replicas, and don't use quorums. Aurora is an instructive counter example (https://lnkd.in/gNrNTPp7).

117

2 Comments

#KubeCon #CloudNativeCon #OpenSourceSummit is coming to an end in Hong Kong, and a few things that I am really proud of #𝗥𝗲𝗱𝗛𝗮𝘁 and our team: 🔷 We were not focusing on "𝗡𝗔𝗠𝗘 𝗖𝗔𝗥𝗗𝗦" and "𝗦𝗔𝗟𝗘𝗦 𝗟𝗘𝗔𝗗𝗦", but instead, more on "𝗦𝗛𝗔𝗥𝗜𝗡𝗚𝗦" and "𝗟𝗘𝗔𝗥𝗡𝗜𝗡𝗚𝗦" 🔷 We didn't see a lot of "𝗖-𝗟𝗘𝗩𝗘𝗟 𝗘𝗫𝗘𝗖𝗨𝗧𝗜𝗩𝗘𝗦" and "𝗠𝗔𝗡𝗔𝗚𝗘𝗥𝗦" in suits, but a lot of "𝗕𝗔𝗖𝗞𝗣𝗔𝗖𝗞𝗦" and "𝗧𝗘𝗖𝗛𝗜𝗘 𝗗𝗥𝗘𝗦𝗦 𝗖𝗢𝗗𝗘𝗦" who were eager to learn 🔷 We didn't see many "𝗖𝗢𝗠𝗣𝗔𝗡𝗬 𝗦𝗣𝗢𝗡𝗦𝗢𝗥𝗦" as speakers who purchased the presentation slots, but more "𝗜𝗡𝗗𝗜𝗩𝗜𝗗𝗨𝗔𝗟 𝗦𝗣𝗘𝗔𝗞𝗘𝗥𝗦" who submitted their own proposals and were selected based on their meritocracy. 🔷 And finally, we didn't see "𝗖𝗢𝗠𝗣𝗘𝗧𝗜𝗧𝗢𝗥𝗦", all we see are "𝗣𝗔𝗥𝗧𝗡𝗘𝗥𝗦" with similar visions This is the Power of Open Source and together, we make the world a more beautiful place. *

91

Watch the final part of the series "Crafting Applications with Cosmos DB" where Saranya Sriram and Sandeep S Nair unpack how surprisingly simple it is to make your app on Cosmos DB resilient in the case of a Datacenter failure. They also refer to a new preview feature that is a steal: Dynamic Autoscaling: https://lnkd.in/e74PXE4V #Autoscale Azure Cosmos DB #HighAvailability #NoSQL #DistributedData https://lnkd.in/e8he9PCy

18

https://lnkd.in/gUpFEbby One of the limitations of current generation #LLM is it's inability to consume non linear data at scale. #SpreadSheetLLM Attempts to get rid of this limitation by focusing on most commonly available non linear data, that is tabular data.

6

1 Comment

Tune in on December 17, as we're diving into the future of table optimization with Beyond Partitioning: Using #LiquidClustering to Escape from Hive-Style! I Say goodbye to the costly guessing game of partition optimization. With liquid clustering, your tables can adapt effortlessly to shifting data volumes—no more stress about getting your partition strategy right from day one. For years, we’ve needed to play an expensive guessing game — will my partition strategy result in too small, too big, or just right file sizes? Do the decisions today still make sense 6 months down the road? What about 1 year? What if you didn’t need to make so many decisions on day one? What if your table could instead adapt to the volume of data with minor effort on your behalf? Welcome to the world of liquid-clustering! In this session you’ll learn to harness the power of adaptive clustering and come to an understanding that today is the day you can finally escape the shackles of hive-style partitioning https://lnkd.in/g_u44AZD

2

At Confluent's Kafka Summit in Bangalore, Shaun Clowes, Chief Product Officer, and Jay Kreps, Co-Founder and CEO, discussed Confluent's strategic approach to enhancing enterprise data infrastructure amidst the AI revolution. "Every time you have a new data need, what do you do? You connect the destination point-to-point to all the different sources needed. Over time, your data infrastructure turns into a giant, tangled mess of interconnections," shared Shaun Clowes. “Kafka often serves as the data feed into this area. However, when considering the various tools available, they generally run on not just streams, but largely on extensive tables of data. This is the essence of a data warehouse or data lake," Jay Kreps explained. #BigData #DataManagement #DigitalTransformation

2