Lubo Karadashkov

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

15

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Comments

💡 Optimizing Costs with Dragonfly: A Technical Breakdown 💻 For technical teams looking to maximize performance while minimizing cloud infrastructure costs, Dragonfly offers significant advantages. By cutting down memory overhead, leveraging CPU-specific optimizations, and using a shared-nothing architecture, Dragonfly is designed to be resource-efficient while maintaining high throughput. Whether you’re scaling horizontally or aiming to optimize performance, Dragonfly can help you reduce operational costs without sacrificing speed. In this blog, we break down the technical strategies that Dragonfly uses to cut costs and improve efficiency. 🔗 Read more here: https://hubs.la/Q02RDNVr0 #CloudOptimization #TechEfficiency #DevOps #DatabasePerformance #Dragonfly#CostReduction

3

How did Dremio take on self-serve analytics with a "lakehouse" 🤔 Hear from Dremio Founder Tomer Shiran on how they made data more accessible through an open source data lakehouse approach using Apache Arrow and Apache Iceberg 💪 And built a $2B company while doing it 😲 Full episode 👉 https://lnkd.in/geQrmn2R

6

Dremio founder, Tomer Shiran, sat down with the Open Source Startup Podcast to talk about how we're making data more accessible through an open source data lakehouse approach using Apache Iceberg. Listen to it here!

7

🚀 Excited to share our latest blog post! "Streaming PostgreSQL Data to kafka Topics using Debezium" In this blog , Brahmini Meruva explains about how to seamlessly integrate PostgreSQL with Kafka using Debezium CDC and Docker. Here is the blog: https://lnkd.in/gvf8b5ek 🚀 Aiming New Postgres Horizons 🚀 Stay tuned !! For more info, reach us at OpenSource DB or https://opensource-db.com/ #Kafka #Data #Datapipeline #debezium #Streaming #logicalreplication #PostgreSQL #OpenSourceDB #osdb

16

Postgres Women India thanks Brahmini Meruva for the blog on "Streaming PostgreSQL Data to Kafka Topics using Debezium" This blog covers the setup of Postgresql,kafka(with kraft),connect(debezium),kafka UI to stream data seamlessly. #postgresql #womenintech #postgres #debezium #CDC #kafka #kafkatopics #docker

1

SqlDBM starting the day strong with data & d🍩nuts! This week we’re preparing for Sn❄wflake Summit in SF - who will we see there? #SnowflakeSummit #data #sqldbm

21

2 Comments

If you ingest data from Kafka/Kinesis into Databricks, you should watch Charles Tan and Shawn Gordon's short vide on how to shift left your data pipelines to reduce cost and latency! #ApacheFlink #Kafka #StreamProcessing #RealTimeData https://lnkd.in/ggMsi9KU

14

1 Comment

🚀 New blog on Postgres write-ahead logs (WAL) We dive deep into the mechanics of Postgres logical replication and the role of WAL. We discuss how WAL works, why you want to use them, and their various components and settings. A few key items we discuss: 👉 How to use WALs to ensure your data remains consistent and protected, even during unexpected failures.  👉 The WALs role in logical replication with change data capture and important configurations to consider.  👉 Explore various WAL settings such as wal_buffers, checkpoint_flush_after, max_replication_slots, and max_slot_wal_keep_size to fine-tune database operations. Losing data is a nightmare for any data engineer. Whether you're a seasoned data engineer or just starting out, this blog is packed with essential insights to help you optimize Postgres for better performance and reliability. Check out the full post below. #postgres #writeaheadlog #dataengineering #datareplication

31

3 Comments

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

I've just added a Terraform configuration for #Confluent Cloud Apache Flink! This sets the stage for upcoming work highlighting the advantages of using Confluent Cloud Apache #Flink over the open-source alternative (such as Amazon Managed Service for Apache Flink). If you're curious how Confluent Cloud Apache Flink stands out, check out this example in action—it's ready to explore!

11

Did you know the 1st database mgmt system was a graph? Listen to this podcast with Semih Salihoğlu about the history of graph databases. It's a facinating evolution of features and as Semi says, "graphs have always existed and will always exist." Teaser: We all need to learn more about Charlie Bachman!

27

2 Comments

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

How can you handle long pauses of your data in #ApacheFlink? Since your stream is stopped, setting an idle timeout won’t work. In the latest video in his Apache Flink in Action series, Dan Weston explores one possible solution. Give it a watch and then share your solution in the comments! Watch it here ➡️ https://lnkd.in/gjmAv46B

50

1 Comment

Handling missing data points is not as easy as it sounds for #streamprocessing "Fancy" mechanisms like "watermark" won't show you new result if the count is 0 for a tumble/hop window. Check the nice video from Confluent. Dan mentioned a solution to add special events in the data stream(say Kafka), for 0USD transactions. Well, it works but will impact your COUNT/SUM/AVG, unless you add many filtering. I would suggest use DataGen or RandomStream in Timeplus as a heartbeat stream, UNION 2 streams together, run TUMBLE window with GROUP BY window and type, then you will get count=0 and fire alerts. SQL like this: WITH cte AS (SELECT 'real' as type FROM stream1 UNION SELECT 'datagen' as type FROM datagen) SELECT window_start, type, count(1) as cnt FROM tumble(cte,1m) GROUP BY window_start, type HAVING cnt=0 Leave comments below if you have any thoughts, or join timeplus.com/slack for more discussions.

8

Exciting news for data enthusiasts and AI practitioners! 🚀 AWS has just released a comprehensive guide on building real-time streaming generative AI applications using cutting-edge services: 📊 Amazon Bedrock 🔄 Amazon Managed Service for Apache Flink 💾 Amazon Kinesis Data Streams This powerful combination allows you to: • Process continuous data streams in real-time • Leverage foundation models from leading AI companies • Perform advanced analytics like sentiment analysis on streaming data The blog post provides a detailed walkthrough, including: • A reference architecture • Step-by-step setup instructions • Sample code using AWS CDK • Practical example: Real-time sentiment analysis of movie reviews Whether you're working with IoT sensors, application logs, or clickstream data, this solution opens up new possibilities for deriving rapid insights from your streaming data.Ready to supercharge your real-time data processing with generative AI? Check out the full article for all the details! 👇 #AWS #GenerativeAI #DataStreaming #ApacheFlink #AmazonBedrock #RealTimeAnalytics

29

Developers can now store, index, and query vectors in PostgreSQL using pgvector, eliminating the need for a dedicated vector database. This blog from Yugabyte expert Brett Hoyer deep dives into pgvector, and explains how to store text embeddings as vectors, execute similarity search, choose between indexes, and scale with distributed SQL.💡 Find out more!⬇️ https://hubs.la/Q02JyYrM0 #YugabyteDB #pgvector #vector #AI #Postgresql #cloudnative #distributedsql

5