Apache Iceberg’s Post

With this move, the largest Cloud provider worldwide blessed #ApacheIceberg as the de-facto standard for "big sql" in the industry So... congrats to #Cloudera on-AWS adopters, as since nearly 3 years (see https://lnkd.in/dRujBd9A) they are already leveraging Apache Iceberg tables persisted in standard s3 buckets... ;) (and with a number of added-value capabilities such as Fine-Grained Access control, automatic metadata harvesting and lineage creation, ...) And, by the way, congrats to #Cloudera on-Azure and Cloudera on-GCP adopters too, as they are equally reading / writing Apache Iceberg tables on-ADLS and on-GCS tables too... And, by the way, congrats to those Data Engineers and Analysts that thanks to #Cloudera's Iceberg REST Catalog (https://lnkd.in/dcVmn8fu), can query Iceberg tables from any Iceberg-friendly MPPs like #AwsAthena, #EMR, #Snowflake MPPs And finally congrats to those Data Strategists adopting #Cloudera Data Platform for its unique ability to run (and move around) their object store-backed Iceberg Tables literally anywhere (including inside their own Data Centres, using #apacheOzone as primary storage) https://lnkd.in/dRvH3545 Yari Franzini Fabio Pascali Sophie Papillon Juan Carlos Sanchez De La Fuente Pablo Boixeda Denis FRAVAL-OLIVIER Ali Bajwa Venkatesh Sellappa Pietro Biasci Réda Kamar Michelle H. Javier Feixas Ballesteros Frank Contrepois Venkat Rajaji Bill Zhang Venkatesh Periyathambi Apache Iceberg

So AWS making glue catalog integration easier for Iceberg - a positive step forward. Iceberg maintenance routines as part of S3 table policies is nice. Still not seeing a graceful solution though to brokering between AWS S3 primary storage (with Glue catalog) - Databricks external tables (with Unity) and Snowflake external tables (with Polaris). Sure you get ACID support allowing for concurrent read/writes, but who is the metadata master? And how do you secure across in a unified way? Anyone? Lakeformation + IAM, Polaris RBAC, Databricks FGAC == a wild ride #opencatalogwars

A day after I read Chris Finlayson's post about the need for a graceful solution to broker between Amazon S3 and external tables in Databricks and/or Snowflake, I came across an article about the Chaos of catalogs on substack which further led me onto a brand new project called Apache XTable (Incubating) that has been brewing since the past few months and seems to have garnered more interest in the community since it was the subject of a talk at the Data+AI summit. The Apache XTable project works with the premise of an open lakehouse by acknowledging that the existence of multiple catalogs in our ecosystem is unavoidable and thus enabling interoperability and seamless integration between these different catalogues such as Iceberg, Hudi and Delta Lake is a more pragmatic approach. To achieve this, XTable will allow you to translate the metadata of one open table format into another, without ever touching the underlying data, thus allowing multiple catalogs to co-exist. This means you no longer have to worry about choosing a catalog or an open table format that is just right for your project. How cool is that? But the thing that has gotten me even more excited is how a project like Apache XTable begs the question, is there really a need to look for a metadata master if you can have multiple catalogs abstracting the same table? The project is far too nascent to know for sure if it can be the answer to all our prayers. But it has definitely got me thinking about the possibilities, because if multiple catalogs can co-exist, it could essentially solve some of the biggest challenges of brokering Amazon S3 with external tables in Databricks and/or Snowflake, such as metadata management (because XTable could be in charge of propagating metadata), query performance (because you could use the most suitable open table format to query the data depending on which query engine you are using) or access control (because XTable could be responsible for managing consistent access control across all formats). The possibilities are endless and exciting and I am really keen to learn more about this project. I think this is definitely a project worth keeping an eye on. If you'd like to see a video of the talk on Apache XTable at the Data+AI Summit or the substack article that I've referenced in this post, here are the links that you can hop on too! https://lnkd.in/dVMMhJZF https://lnkd.in/dZzibhsk https://lnkd.in/drQD5ftw #datalakehouse #dataengineering #opentableformats #opencatalog #iceberg #hudi #deltalake #xtable #onehouse #womenindata #womenintechnology #womenintech #dataaisummit #substack #openlakehouse

Let’s start test it

One more step towards making Iceberg the overall winner of the most popular open table formats out there! We could see this coming for a while now (remember the recent Databricks purchase of Tabular, for example?) but this is, in no uncertain way, another massive step towards Iceberg making it all the way to the top within all the big public cloud providers when considering open table formats to support modern data platform solutions. I'd expect Azure and GCP to follow suit in their efforts to keep facilitating, easing (and even defaulting?) the adoption of Iceberg natively within their data lakehouse service implementations and related ecosystems #iceberg #lakehouse #dataplatform

In 2024, the choice of datalake table format become clearer - #Iceberg. Big data players has put their resouces on #Iceberg, rather than #hudi and #deltalake.

🚀 AWS Glue Data Catalog now supports automatic optimization of Apache Iceberg tables that can be only accessed from a specific Amazon Virtual Private Cloud (VPC) environment. ☁️ Customers who have governance and security configurations that require an Amazon S3 bucket to reside within a specific VPC can now use it with Glue Catalog. 💾 👉 Automatically compact Iceberg tables, remove unused data & metadata, reduce storage costs & improve query performance while keeping your tables secure. 👨🏽💻 Read the blog now to learn how to optimize your Iceberg tables regardless of where it's stored on Amazon S3 with just a few clicks! https://lnkd.in/gTQajd-r #AWSGlue #AmazonS3 #ApacheIceberg

Hello family!!! “TASK-1” Few days back, I got a feed like, In AWS, I have 2 instances, pne is public and other private with NAT gateway enabled. In that private instance, I have an HTML content. Now I want to make visible of the private content using public ip with the help of BATION HOST and Port forwarding techniques. Can any one got an idea, input about this task, leave your comments. Will discuss there!!! #AWS #cloud #VPC #EC2

https://lnkd.in/gbVSPbNv #Tool for discovering the origin host behind a #reverse #proxy. Useful for bypassing #cloud WAFs! #waf