Data Flow Digest #2 - More Databases?!

We're back! This week, we'll take a quick look at the new database (to rule them all) from Amazon and some cool new releases from Flow, and we'll close the week off with interesting blog recommendations for the weekend.

Spotlight: dSQL

AWS re:invent is over, and oh boy, Christmas came early! Last week, we looked into S3 Tables, which is Amazon's take on a managed Apache Iceberg lakehouse, but that wasn't the only interesting thing announced!

Amazon Aurora DSQL is AWS’s latest serverless, distributed SQL database, delivering 4x faster reads and writes than competitors and 99.999% multi-region availability. Built on PostgreSQL, it ensures strong consistency and seamless integration with existing tools, making it ideal for high-demand industries like finance, gaming, and e-commerce.

Key features include:

• PostgreSQL Compatibility: Easy migration and familiar tooling.
• Serverless Design: Automatic scaling for compute, I/O, and storage.
• Active-Active Architecture: Resilient global applications without sharding.

While its AWS-only deployment may lead to vendor lock-in, Aurora DSQL’s performance and scalability set a high bar for distributed SQL databases.

Explore how it works by reading the docs.

Enterprise

Gartner just released their new version of the data integration magic quadrant. As usual, it has caused quite a commotion in the dataengineering subreddit.

Release Radar

Let's take a look at some early presents that the Estuary team has been working on this week.

1. Default schema names: Flow will now infer schema names from coming from your source database and use those as the default when creating new datasets in the destination. No more messy datasets in your warehouse; time to get organized!
2. Source BrainTree: We just shipped a native, real-time BrainTree connector. Say goodbye to batch exports and hello to instant visibility!
3. Materialize Kafka: New connector for materializing data to Kafka topics. It can encode messages as JSON, or Avro when there is a schema registry configured!

Did You Know?

When you backfill a capture in Flow that is linked to a materialization, you can choose to trigger a backfill for the linked connector, which can save you a lot of time! Check it out 👇

Something Interesting

To close off the week and start getting into that holiday mood, check out these cool articles that we recently published on our blog (click the images to read the full articles):

Iceberg Catalog Showdown

A great deep dive into the main differences between the two catalogs. Article by Karen Zhang .

Adding Sentiment Analysis to Survey Results

A hands-on tutorial that shows you how you can build a real-time sentiment analysis data flow! Article by Emily L.