Julien SIMON

📢 New Video Alert: Keynote by Julien Le Dem on Apache Iceberg and the Deconstructed Database 🎥 Watch Julien's keynote from the #ChillDataSummit in #SanFrancisco. Julien Le Dem is one of the key innovators in the open-source community and a driving force behind projects like Apache Parquet. In this talk, Julien explores how Apache Iceberg is revolutionizing data lakes, breaking down the concept of the deconstructed database and its impact on large-scale data management. If you're passionate about data architecture or curious about the future of data processing, don’t miss Julien’s insights on how Iceberg is pushing the boundaries of what's possible in data lakes. 📽️ Watch the full video here: https://lnkd.in/d9eg2exq #ApacheIceberg #OpenSource #DataEngineering #DataLakes #BigData

26

1 commentaire

AutoMQ: Advancing Event Streaming Beyond Apache Kafka I've been exploring AutoMQ, a new event streaming platform that's fully compatible with Apache Kafka but addresses many of Kafka's limitations. Here's what you need to know: 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗼𝗳 𝗔𝘂𝘁𝗼𝗠𝗤: ✅ 100% Apache Kafka compatible ✅  10x more cost-effective ✅  Instant auto-scaling capability ✅  Single-digit millisecond latency ✅  S3-based storage layer 𝗔𝘂𝘁𝗼𝗠𝗤 𝘀𝗼𝗹𝘃𝗲𝘀 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗞𝗮𝗳𝗸𝗮 𝗽𝗮𝗶𝗻 𝗽𝗼𝗶𝗻𝘁𝘀: 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀: Kafka's scaling involves time-consuming partition reassignment. AutoMQ uses stateless brokers for instant scaling. 𝗗𝗮𝘁𝗮 𝗦𝗸𝗲𝘄: Kafka's static partition distribution leads to skew. AutoMQ implements built-in self-balancing. 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗖𝗼𝘀𝘁𝘀: Kafka relies on expensive local SSDs. AutoMQ leverages cost-effective S3 storage. 𝗛𝗼𝘁/𝗖𝗼𝗹𝗱 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗲𝗻𝘁𝗶𝗼𝗻: AutoMQ separates hot and cold data access for optimized performance. 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗖𝗹𝗼𝘂𝗱-𝗡𝗮𝘁𝗶𝘃𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁: AutoMQ's architecture is designed for cloud environments and K8s. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀: 𝗦𝘁𝗮𝘁𝗲𝗹𝗲𝘀𝘀 𝗕𝗿𝗼𝗸𝗲𝗿𝘀: Enable rapid scaling without data replication. 𝗦𝗲𝗹𝗳-𝗕𝗮𝗹𝗮𝗻𝗰𝗶𝗻𝗴: Automatically migrates partitions based on throughput. 𝗦𝗵𝗮𝗿𝗲𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: Utilizes cloud storage services for durability and performance. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗔𝗰𝗰𝗲𝘀𝘀: Employs pre-reading, concurrency, and Direct I/O for superior performance. AutoMQ offers a compelling solution for teams looking to enhance their event streaming infrastructure without disrupting existing Kafka setups. Its architecture addresses scalability, cost-efficiency, and performance challenges inherent in traditional Kafka deployments. Interested in a deeper technical dive ? Here is the official website - https://www.automq.com/ Github Repository: https://lnkd.in/e5f2u7BF

326

19 commentaires

I'm very happy to share a technical article I co-created with Deeksha Gunde, that walks through enabling a federated Airflow environment using AWS SNS/SQS. https://lnkd.in/e4RwPirp #aws #dataengineering #analytics #bigdata

33

⏰ Check out our new blog post : “Chunking for RAG: best practices”. Chunking is a crucial data preprocessing step for RAG systems, but finding the optimal approach can be complex. In this post, we talk about the reasons for chunking data, explore common methods, and highlight Unstructured smart chunking strategies that preserve the semantic structure of your documents. These strategies ensure uninterrupted information flow and enhance retrieval precision, improving your RAG systems’ performance. Key takeaways: - Understanding the importance of chunking for RAG - Common chunking methods - Unstructured's smart chunking strategies - Practical tips for chunking optimization Get started for FREE with our Serverless API and bring your RAG system's performance to the next level. We can't wait to see what you build! Read the full blog post here: https://lnkd.in/eGNvyuJz

57

2 commentaires

Data Engineers ! This one's for you: 🧠 The Economics of Generative AI Stephanie Kirmer 🧠 Connecting ML Models in Sagemaker to Snowflake and dbt Hugo Lu 🧠 Introduction to Interpretable Clustering NAKUL UPADHYAY 🧠 Snowflake automatic clustering cost estimation Tim Sander 🧠 Death to Tech Debt? Ari Joury, PhD Check it here; https://lnkd.in/ebdFKNZ5 #dataengineering #dataroundup #startup

5

Apache Hive is a data warehouse open source project to provide data querying and analysis capabilities supoorting SQL. The new release includes several features to leverage Iceberg: - Advanced Snapshot management - Branches & Tags support - DML (insert/update/delete/merge) - COW & MOR modes - Vectorised Reads & Writes - Table migration command - LOAD DATA statements support - Partition-level operations support - Improved statistics (column stats support)

49

3 commentaires

A little preview of the next big release of Apache Doris 👀 : It will support the compute-storage decoupled mode. #cloudcompute #database #dataengineering #opensource #S3 #HDFS #MinIO

16

Dive into the benefits, architecture, and practical implementation of a feature store at runtime in our latest article from Alexandre AUTRET, data scientist at Numberly: https://nubr.ly/LNvJxL You can also stay updated on all our latest posts by following our Numberly Tech Medium channel here: https://numberly.tech/ #MachineLearning #DataScience #FeatureStore

5

📢 Join us at LIVE in ONE HOUR for the next YugabyteDB Community Open Hours. In this episode, Yugabyte experts Franck Pachot and Denis Magda will dive into building scalable and fault-tolerant multi-tenant applications using distributed PostgreSQL (YugabyteDB).💡 ➡️ The team will start with the basics of multi-tenancy in single-server PostgreSQL deployments. ➡️ Next, they'll detail how to use sharding to distribute and isolate tenants across multiple database servers. ➡️ Finally, they'll demonstrate how to simplify multi-tenant architectures by eliminating single points of failure and automatically distributing tenants across zones, regions, and database nodes. Bring your questions to the chat and join the discussion.🚀 Register now!➡️ https://hubs.la/Q02KW_c10 Join the YugabyteDB community!➡️ https://hubs.la/Q02KWYKW0 #YugabyteDB #cloudnative #distributedsql #Postgresql #community #opensource #sharding

3

Spotting Postgres performance issues with Coroot 💡 🔗 https://lnkd.in/eUEP-EXH 🧑‍💻 Nikolay Sivko : "We spot a spike in the catalog‘s latency. To determine whether this anomaly is caused by a Postgres performance issue, we need to somehow compare the app latency with the DB latency. If these metrics correlate we can go deeper to find the reason for the degraded database performance."

6

⏰Join us at 11am ET TOMORROW for the next YugabyteDB Community Open Hours.⏰ In this episode, Yugabyte experts Franck Pachot and Denis Magda will participate in the recently launched Postgres App Century Challenge by performing a live migration of several existing Postgres applications to YugabyteDB. 👀 You’ll learn migration tips and tricks, plus how to avoid pitfalls. Join us to see how many apps Franck and Denis can migrate in one hour!💡 Learn more about the challenge here: https://hubs.la/Q02M_Mn_0 Bring your questions to the chat and join the discussion.🚀 Register now!➡️https://hubs.la/Q02M_NMS0 Join the YugabyteDB community!➡️ https://hubs.la/Q02M_GVT0 #YugabyteDB #cloudnative #distributedsql #Postgresql #community #opensource #migration

3

🚨 Looking for a powerful alternative to Kafka? ApachePulsar's unique architecture sets it apart in building event-driven applications. Watch Julien Jakubowski 🥑's #OSACon 2023 session and learn about its scalable processing power, exceptional message durability, and comprehensive use case coverage, rivaling #RabbitMQ and #Kafka with a single broker. #opensource https://lnkd.in/gMt72TUr

11

Postgres has a highly extensible architecture, but it needs to be modified to support large-scale analytics in order to handle the AI workloads of the future. That was one of the key topics at hand as EDB Senior Director Franck Sidi sat down with EDB Technical Fellow Dr. Marc Linster to discuss the relationship between AI and analytics, as well as how to apply AI techniques to enhance or automate analytics. In order to support large-scale analytics, Postgres has to be extended in these four ways: 1. Decoupling compute and storage - Decoupling the compute engine with the storage engine to allow independent scalability. 2. Dynamic scalability - Horizontal and dynamic scalability for compute with a distributed query engine. The same query can be split and run on multiple computer nodes using a distributed query execution engine. 3. Columnar storage - Support for Columnar Storage format for the tables is essential for query execution speed to access only relevant columns. This also enables efficient data compression as you can apply compression algorithms per column. 4. Vectorization - A vectorization query engine makes it possible to leverage the CPU modern chipset. Vectorization means that you can work on multiple rows in 1 CPU instruction. Single Instruction, Multiple Data (SIMD concept) Want to dive deeper into using Postgres for analytics, search, and AI? Read the full blog here ➡️ https://bit.ly/3ZsbYYL #EDBPostgresAI #JustSolveItWithPostgres #AIAnalytics #LargeScaleAnalytics #PostgreSQL #VectorDatabase

34

1 commentaire

Monday Morning ☕ calls for Orchestra's data roundup. It is also swiflty becoming a roundup of roundups too I guess as we like to include Christophe Blefari's newsletter in ours??? Check it out here ⚡ https://lnkd.in/ezV_wfVr 🧠 Stop being data-driven Cai Parry-Jones 🧠 Why did Databricks open-source Unity Catalog 🧠 7 surprising habits that make data engineers unstoppable Tim Webster 🧠 How to run Kafka on 100% object storage Vu Trinh 🎥 How to build a data infrastructure that lasts Benjamin Rogojan Seattle Data Guy and great content from Joe Reis 🤓 as always #dataengineering #dataroundup #datanews

14

3 commentaires

New post! I talk with Philippe Noël about ParadeDB, the experience of building as a PostgreSQL extension, pg_duckdb, pg_lakehouse, and more...

25

2 commentaires

When I read this first, I was like, Tools updated for MySQL 10.1 ? That has not been released yet... https://lnkd.in/e9EDWHWq #mysql

1

Good energy here, really appreciated this meetup! 🤩 Look at the recording and recognize the common mistakes with Kafka (the audience definitely did!). How to handle invalid messages, manage schema evolution, and ensure data quality. It’s a tough task, whether you’re using CI/CD, a custom SDK, or a Kafka proxy. #meetup #apachekafka

5