Apache Iceberg

❄️ Iceberg Ahead! 🚢 I just dropped a fresh newsletter diving into Apache Iceberg—the ultimate upgrade for your data lakehouse! Here’s a sneak peek: ✅ File skipping = 3x faster queries (skip the gym, your data’s got it). ✅ WAP workflows—Git for your data (all correct data or none). ✅ Governance, security & the 🍒 on your OLAP cake 🎂. 💌 Subscribe now : https://lnkd.in/emrkZNBx — Ameena 💻✨ #DataEngineering #ApacheIceberg #DataLakehouse 👉 Read here : https://lnkd.in/eXHv7KKH

🚀 Exploring Data Architectures: Data Lake, Warehouse, and Lakehouse 🚀 #DataWarehouse #DataLake #Lakehouse https://lnkd.in/dad5Qn9K

¿𝗤𝘂é 𝘀𝗼𝗻 𝗹𝗼𝘀 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗖𝗮𝘁𝗮𝗹𝗼𝗴𝘀 𝘆 𝗽𝗼𝗿 𝗾𝘂é 𝘀𝗼𝗻 𝘁𝗮𝗻 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁𝗲𝘀 𝗲𝗻 𝗹𝗮𝘀 𝗽𝗹𝗮𝘁𝗮𝗳𝗼𝗿𝗺𝗮𝘀 𝗱𝗲 𝗱𝗮𝘁𝗼𝘀 𝗵𝗼𝘆 𝗲𝗻 𝗱í𝗮? 📍 𝗨𝗻 𝗰𝗮𝘁á𝗹𝗼𝗴𝗼 𝗱𝗲 𝗱𝗮𝘁𝗼𝘀 𝗲𝘀 𝘂𝗻 𝗿𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝗶𝗼 𝗼 𝗶𝗻𝘃𝗲𝗻𝘁𝗮𝗿𝗶𝗼 𝗱𝗲 𝗹𝗼𝘀 𝗮𝗰𝘁𝗶𝘃𝗼𝘀 𝗱𝗲 𝗱𝗮𝘁𝗼𝘀 𝗱𝗲 𝘂𝗻𝗮 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝗰𝗶ó𝗻. 𝗘𝘀𝘁𝗼𝘀 𝘂𝘁𝗶𝗹𝗶𝘇𝗮𝗻 𝗹𝗼𝘀 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗼𝘀 (técnicos, descriptivos, operativos o de gobernanza) 𝗽𝗮𝗿𝗮 𝗮𝗱𝗺𝗶𝗻𝗶𝘀𝘁𝗿𝗮𝗿 𝗹𝗼𝘀 𝗱𝗮𝘁𝗼𝘀. Hoy en día las grandes empresas tienen ecosistemas de datos amplios y complejos. Algunas de las características que deben soportar estos catálogos son las siguientes: ✅ Gestión de metadatos. ✅ Descubrimiento y perfilado de datos. ✅ Linaje de datos. ✅ Gobernanza. ✅ UI/UX. ✅ Control de acceso. ✅ Calidad. ✅ Observabilidad. ✅ Integraciones. 📍 Durante los 2 últimos años, muchas empresas que trabajan con datos han desarrollado o mejorado sus catálogos de datos. En el ámbito de los Data Lakehouses, y con la llegada de los OTFs (Iceberg, Delta Lake y Hudi), podemos clasificarlos de la siguiente manera: 1️⃣ 𝗢𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗱𝗮𝘁𝗮 𝗰𝗮𝘁𝗮𝗹𝗼𝗴𝘀: Son proyectos mantenidos por la comunidad (no vendor lock-in), suelen ser gratuitos y pueden ampliarse con nuevas funcionalidades. No suelen contar con soporte oficial. e.g: #ApachePolaris, #ApacheGravitino, #UnityCatalog, #Nessie, #DataHub o #OpenMetafata. 2️⃣ 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 (𝗖𝗹𝗼𝘂𝗱-𝗯𝗮𝘀𝗲𝗱) 𝗱𝗮𝘁𝗮 𝗰𝗮𝘁𝗮𝗹𝗼𝗴𝘀: Garantizan un mejor acceso distribuído, mejor seguridad y reducen gastos de gestión. Pueden incluir funcionalidades adicionales (normalmente a partir de la versión OSS) y ofrecen diferentes opciones de despliegue: Managed (Software as a Service), Self-deployed, etc. Precio muy variable. e.g: #SnowflakeHorizon, #SnowflakeOpenCatalog, #DatabricksUnityCatalog, #FabricOneLakeCatalog, #GoogleDataplexCatalog, #AWSGlueCatalog o #MicrosoftPurview. 🔝 Apache Iceberg, a día de hoy, es el OTF más popular en los Data Lakehouses. Soporta 2 tipos de catálogos: File-based Catalogs y Service-Based Catalogs. 1️⃣ 𝗙𝗶𝗹𝗲-𝗯𝗮𝘀𝗲𝗱 𝗖𝗮𝘁𝗮𝗹𝗼𝗴𝘀: Estos catálogos funcionan en cualquier sistema de ficheros, ya sea Hadoop o object storages. Mantienen la referencia al último metadata.json. También conocidos como Hadoop Catalog. 2️⃣ 𝗦𝗲𝗿𝘃𝗶𝗰𝗲-𝗕𝗮𝘀𝗲𝗱 𝗖𝗮𝘁𝗮𝗹𝗼𝗴𝘀: Basados en un servicio o servidor. Los motores de consulta obtienen el fichero metadata.json en base al servicio. e.g: Hive Metastore Catalog, JBDC Catalog, AWS Glue Catalog o Nessie Catalog. 🔥🔥 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 𝗥𝗘𝗦𝗧 𝗖𝗮𝘁𝗮𝗹𝗼𝗴 se está convirtiendo en un estándar de una especificación de catálogo REST, permitiendo la interoperabilidad entre catálogos o plataformas. ¿Quienes lo soportan? Apache Gravitino, Apache Polaris, Nessie, Unity Catalog, Snowflake, Databricks... #DataCatalog #DataManagement #IcebergRESTCatalog #DataLakehouses #DigitalTransformation #DataGovernance #DataEngineering

Open Lakehouse Architecture - But what's really open? There’s a noticeable shift in how users/customers are now consistently thinking about data architectures. You hear about terms like 'Open lakehouse'. But what does “open” really mean? This is not something easy to define. Still, it looks like there is a general agreement on one core idea. Data should reside as an 'open and independent' tier, allowing all compatible compute engines to operate on that “single copy” based on workloads. The most important thing to understand here is that just replacing proprietary storage formats with 'open table formats' doesn’t automatically make everything open and interoperable. In reality, customers end up choosing a particular open table format (based on vendor support), while being tied to proprietary services & tools for things like optimization, maintenance, among others. This confusion is created by the growing use of jargons like "open data lakehouse" and "open table formats”. And no, this is not about build vs buy! You can still buy vendor solutions while maintaining an open & interoperable platform. The 'key' is that when new workloads arise, you should be able to integrate other tools or seamlessly switch between compute platforms. I sought out to answer some of the questions that have been in my mind in this blog (in comments). This is from the perspective of having worked with the 3 table formats (Apache Hudi, Apache Iceberg & Delta Lake) for the past couple years of my career. Some questions that I ask: ✅ What are the differences between an open table format & an open data lakehouse platform? ✅ Is an open table format enough to realize a truly open data architecture? ✅ How seamlessly can we move across different platforms today? Would love to hear any thoughts! #dataengineering #softwareengineering

Apache Iceberg and Trino are revolutionizing enterprise data lake architectures by tackling cost, scalability, and interoperability challenges. 🔑 Key Highlights from SiliconANGLE: - Iceberg delivers database-like functionality on object stores for seamless, cross-platform analytics. - Trino, co-created by Starburst’s Dain Sundstrom, powers fast, distributed query performance. - Together, they fuel innovation in modern data stacks like Starburst’s Icehouse, a fully managed Iceberg and Trino data lake. 💡 Explore how Starburst is advancing modern data strategies: https://okt.to/JVh6tB #Trino #Iceberg #DataInnovation #OpenSource

📊 Apache Iceberg Tables in Snowflake! 📊 In this brief presentation, I’ve compiled key information about the Apache Iceberg functionality, including main features, setup, and integrations. You’ll find answers to questions like: - What is Apache Iceberg? and its role in data management - Key features, such as ACID transactions, schema evolution, and time-based snapshots - Integrating and setting up Iceberg in Snowflake - Use cases and limitations of Iceberg If you're interested in learning more about this tool or its capabilities - there is cool Tutorial: Create your first Apache Iceberg table 👇 https://lnkd.in/dStt-cnJ #ApacheIceberg #Snowflake #DataLake #DataEngineering #CloudStorage #Tech #DataManagement

📢 GET READY for Iceberg Streaming Analytics Meetup on November 21st🚀 Join RisingWave, MotherDuck and Confluent for an evening packed with the latest on streaming analytics, Apache Iceberg and modern data architectures ⚙️ 💡Agenda: 5:30 PM: Doors open! Grab some food, drink, and mingle 6:00 PM: Kafka Meets Iceberg – Real-time streaming into modern data lakes by Kasun Indrasiri ⚡ 6:30 PM: Iceberg + Postgres Protocol – RisingWave’s Iceberg power-up by Yingjun Wu 🌊 7:00 PM: When All You Have is a Hammer: Using SQL in Your Data Lake by Jacob Matson ⚒️ 7:30 PM: Network and hang out with fellow data nerds! 📍 Location: MotherDuck HQ, 2811 Fairview Ave E, Suite 2000, Seattle 🎟️Don’t miss out! RSVP now 👉 https://lu.ma/1xokjeia #DataStreaming #StreamProcesing #RealTimeAnalytics #SQL #Kafka