Ashish Dubey

Take that Azure Cosmos DB powered app to prod with the help from Sandeep and Saranya! Watch the three part video series: "Mastering #AzureCosmosDB: A Comprehensive Guide from Prototype to Production." https://lnkd.in/eEMt5TFG

1

In #PostgreSQL, global indexes use a shared-nothing architecture to distribute index data across multiple nodes or shards, leveraging table partitioning. PostgreSQL enables parallel processing and enhances performance for large-scale workloads by indexing partitioned tables based on a partition key. #PostgreSQLPerformance @MinervaDB #dba https://zurl.co/a4zF

3

It gives us great pleasure to announce that Apache Spark in BigQuery is now officially available in Private Preview. This feature expands BigQuery’s processing capabilities by offering Apache Spark as an OSS processing engine under a unified control plane (admin, monitoring, billing, security, governance). It is the first step in delivering on the Unified Data Platform vision and was a key announcement at Next ‘24 #googlecloud #bigquery #spark

38

2 Comments

Unleash the power of Postgres for analytics with DuckDB integration. Supercharge your data workloads by seamlessly integrating Postgres with embedded fast query engines. Learn more: https://lnkd.in/g5BppwmB

24

Curious how to use Apache Hudi & Iceberg tables in Databricks environment? Check out this hands-on blog by Dipankar Mazumdar, M.Sc 🥑 that shows how to use Apache XTable to interoperate between the 3 table formats in a lakehouse architecture. 🔗 Link: https://lnkd.in/dwTdBinz #dataengineering #softwareengineering

17

1 Comment

A really nice read from Sai Maddali on the benefits of Tiered Storage with Amazon MSK. https://lnkd.in/gGNJ5nyt #ApacheKafka #Kafka #AWS #MSK #tieredstorage #streaming #storage

55

2 Comments

Amazon's Managed Service for Apache Flink has rolled out an updated connector for Amazon Kinesis Data Streams: https://lnkd.in/gJWEB25j #streamingdata #connectors #AWS #ApacheFlink #Kinesis #data

26

🤔 What are the differences between Data Fabric & Data Mesh? How does Confluent enable these architectures? 🎙️ We answer these questions and more in Episode #4 of our Confluent Capabilities 'Bite Size' podcast series with Italo Nesi and John Dee! Discover how Confluent enhances both of these models through its cloud platform, showcasing it as a versatile solution and demonstrating its crucial role in modern data strategies. 🖥️ Watch the latest episode here: https://lnkd.in/emG9jRJ2 Paul Archer Joanna Grimmond Chris Gallagher Bob Feeney #HybridCloud #MultiCloud #Confluent #DataFabric #DataMesh

10

#ApacheHudi is designed as a specialized storage layer, offering advanced features like indexing, change streams, incremental queries, and log merges to power incremental data processing. In this #InfoQ video, Saketh Chintapalli & Bhavani Sudha Saktheeswaran share real-world experiences gained from operating #Uber’s critical data warehousing pipelines on the new incremental model leveraging Apache Hudi, along with results from the OSS community. 👀 Watch now: https://bit.ly/4dTSRe7 #transcript included #DataEngineering #BigData #DataWarehousing #IncrementalProcessing

14

1 Comment

In case you missed it, we have now released the second issue of our Confluent Chronicles – a fun and entertaining take on the fundamentals of stream processing, Apache Flink, and Confluent! We had such great feedback on the first issue that we couldn’t just stop there – you can check out and download the latest edition here: https://lnkd.in/gEFxA7af Would love to hear what you think of it!   #ApacheFlink #ApacheKafka #Confluent

39

1 Comment

Learn more about how Gemini in BigQuery helps bring the power of AI to your Data Warehouse! https://lnkd.in/d9digzES

13

1 Comment

How can you handle long pauses of your data in #ApacheFlink? Since your stream is stopped, setting an idle timeout won’t work. In the latest video in his Apache Flink in Action series, Dan Weston explores one possible solution. Give it a watch and then share your solution in the comments!

2

In this tutorial we’ll present a CDC pipeline with Kafka to stream changes to a Snowflake Lakehouse in Real Time

5

I'm really excited to share this new blog on Amazon's Exabyte-Scale migration from #Spark to #Ray! 🐘 1.5EiB of data processed in a quarter 🚀 82% better cost efficiency 💸 $120M saving per year OK you might ask: isn't #BigData processing Apache Spark's bread and butter? 🤔 Well, this is a story about software abstractions.. Like Patrick Ames wrote in the blog, "... (Amazon engineers) had limited options to resolve performance issues due to Apache Spark successfully (and unfortunately in this case) abstracting away most of the low-level data processing details". So, the takeaway: if you need more flexibility in data processing (e.g. need GPU, or #unstructured data like video), let's talk! https://lnkd.in/gesSyxHF to get started https://lnkd.in/gfj2Xp_2

205

5 Comments

Its just amazing how good Amazon's internal data systems are and how far ahead they are from the industry in general. They implemented a lot of things at Petabyte scale even before the industry started talking about it. An amazing read! Brings back memories of our own Oracle migration when we were one of the first teams to offboard Oracle within Amazon Ankur sinha Yogesh Kansal #amazon #aws #redshift #data #dataengineering

45

10 Comments

How Uber served 40 million reads with an integrated Redis cache? 🚀 Well, at GigaSpaces Technologies we could achieve this in a fraction of the time/effort and support 10x the load without stressing over cache invalidation, cross-region cache warming, or sharding complexities. Why? Because with our in-memory distributed data grid (XAP Skyline), real-time processing, scalability, and multi-region capabilities are built-in features. Forget about managing separate cache layers and invalidation logic—our architecture eliminates the bottlenecks, giving you ‘consistent, low-latency access’ to data, even under peak load. Uber Engineering and anyone reading this - If you’re tired of wrestling with cache misses, sharding schemes, and overloading databases, maybe it's time to consider a solution that doesn’t just *scale*—it flies. ✈️ #datagrid #scalability #cacheless

3

Simplify your JSON data management with Kipi Data Utils Dynamic JSON Flattener for Snowflake! ❄️ Flatten JSON data, adapt to schema changes, and extract precise data elements effortlessly. Gain deep insights, customize data extraction, and maintain control with built-in versioning and rollback. Try it out today on the Snowflake Marketplace! https://lnkd.in/dpUHBMDA #JSONManagement #DataUtils #Snowflake #DataHandling

7