Cloud-Native Geospatial Forum (CNG)’s Post

Finally finished up my post on building Flexible Data Schemas, see https://lnkd.in/dM-jj5Ye This one was in 'draft' for awhile - I meant to finish it before the TGE workshop that kicked off the field boundary initiative (see https://lnkd.in/gQUMVUDM). The plan was to try to lay the groundwork for some things to explore with data schemas in field boundaries, and also call attention to the great work Overture Maps Foundation is doing with schemas. But we got too busy with workshop prep, and then Matthias Mohr ended up building many of the 'ideas' from the post, adapting several STAC approaches to work for more general vector in the fiboa project (https://github.org/fiboa). And Overture has been continuing to evolve their data schemas in great directions. Matthias just met with Jacob Wasserman and I think we'll see some more collaboration between projects and approaches. I believe it'll lay a foundation for any domain to take a 'bottom up' approach to defining and evolving their data schemas with Cloud-Native Geospatial Foundation formats. This post is not written to appeal to a wide audience, but I appreciate the attention of anyone wanting to work together to make geospatial data more accessible and interoperable. Not just at the 'format' level, but getting to true interoperability one level below, in a virtuous cycle with data and software as I discussed in https://lnkd.in/dTKVkEuQ

Data Catalogs are getting much-needed attention across #datalakehouse and #datawarehouse as the plot thickens, as they say. We are sharing some of the deep internal research we did to support our multi-catalog sync feature in the Onehouse product in this blog from Kyle Weller. https://lnkd.in/giZfaJXH A few callouts… ✅ Tying data formats or table formats open/closed tightly with catalogs is both an anti-pattern and a recipe for lock-in pain. E.g., Hive metastore supported multiple formats. Props here to Unity Catalog and Apache Gravitino, starting on solid technical foundations with cross-format support. 🏗️ Not so long ago, we built a data stack live by assembling all the different “lego” pieces, including different catalogs. If you are interested in a deeper dive into how everything fits, all the way from files on s3 to tracking #datalineage across all your data estate, check this out https://lnkd.in/gtQHJr72 💡 The budding interest in new Open APIs to replace HMS is excellent, but this does not mean a single catalog across all engines. For the foreseeable future or even forever, each engine will run its own catalog with support for open APIs. Physics. 💪 For the open data ecosystem to thrive, support for multiple catalogs is a must and support for syncing to multiple catalogs as a core table service in #apachehudi has been a proven success for the community at large. https://lnkd.in/gNs_tUma  #datacatalog #unitycatalog #datahub #apachepolaris #atlan #awsglue #apachegravitino #dataengineering #apachehudi #apacheiceberg #deltalake

It's a little-known feature, but Snowflake allows you to perform change data capture (CDC) using "streams". This enables the implementation of some really cool use cases, such as RAG, in an easy way. I wrote an article about how you can set up such a project from scratch using Pinecone and Estuary in just a few minutes. Link to repo with all the code in the comments ⬇️ https://lnkd.in/gF2ngwQ5 #dataengineering #snowflakesummit

A semantic layer is an essential component in data analytics that serves as an intermediary layer between the raw data stored in databases and the end-user applications or data visualization tools. It translates complex data structures, relationships, and technical schema names into a business-friendly format that is understandable and accessible to non-technical users. This translation allows end-users to explore, manipulate, and analyze data without needing to understand the underlying data sources or query languages. Cube is one of the best open-source analytical API platforms that provides a semantic layer for every data app. #dataanalytics #semanticlayer #datascience Github: https://lnkd.in/dkWpNa4n

If you are building AI applications on top of #Snowflake, check out this in-depth tutorial by Daniel Palma to learn how easy it is to build a CDC data flow that feeds a Pinecone database with vector embeddings in near real-time that you can use as a datastore to power a ChatGPT-like app.

**Choosing the Right File Format for Storing Big Data: Best Practices and Considerations** In the world of Big Data, the file format you choose can make or break your data processing workflows. With numerous formats available, each suited to different types of data and use cases, making an informed choice is essential. Here's a quick guide to help you navigate this critical decision: **1. CSV:** - **Best for:** Simplicity and interoperability. - **Pros:** Easy to read and write, supported by nearly all data processing tools. - **Cons:** Lack of schema, can be inefficient for large datasets. **2. JSON:** - **Best for:** Semi-structured data. - **Pros:** Human-readable, flexible schema, widely used in web applications. - **Cons:** Can be verbose, not ideal for very large datasets. **3. Parquet:** - **Best for:** Columnar storage for analytical queries. - **Pros:** Highly efficient for read-heavy operations, supports complex nested data structures, better compression. - **Cons:** More complex to write, not as human-readable. **4. Avro:** - **Best for:** Row-based storage and data serialization. - **Pros:** Supports rich data structures, schema evolution, efficient serialization. - **Cons:** Requires a schema definition, less human-readable. **Key Considerations:** - **Data Volume and Velocity:** Choose formats that handle large volumes and high-speed ingestion efficiently. - **Query Patterns:** Consider whether your queries are read-heavy or write-heavy. - **Interoperability:** Ensure the format is supported by your existing tools and platforms. - **Compression and Efficiency:** Look for formats that offer good compression and minimize storage costs. Selecting the right file format is crucial for optimizing performance, cost, and maintainability in your Big Data projects. What file formats do you prefer, and why? Share your experiences in the comments! #BigData #DataStorage #DataScience #DataEngineering #FileFormats #CSV #JSON #Parquet #Avro Tagging for better reach Karthik K.

It's easy to focus solely on data movement, potentially neglecting data usability. A critical yet often overlooked aspect is metadata. Metadata, which provides information about your data, is essential for maintaining context, accuracy, and integrity. In this blog Bruce and I collaborate to demonstrate how to establish and manage metadata as data flows through ArcGIS, particularly in scenarios involving ETL processes where ArcGIS Data Interoperability is used. https://lnkd.in/eGk8F2Dz

Looking to create a realtime data platform? I recently wrote a post on how to create one using Pinot, Trino, Debezium, and Airflow. It’s a simple architecture that allows deriving insights from data quickly. The added benefit is that it’s designed to be self-service so that the data science and analytics teams can move independently.

Check out our latest Pinterest engineering blog post where we unveil the innovative Structured Datastore (SDS) – our multi-model data management solution designed to streamline data processing and create a one-stop-shop for multi-model online query processing. 📌 Explore how SDS integrates a unified serving stack, enabling our teams to efficiently manage diverse data models while improving application scalability and reliability. https://lnkd.in/eUpzAHE7

The DataFusion SIGMOD paper is officially out. Congrats to Andrew Lamb and the other authors! The paper is great, accessible read covering the architecture of a modern, state-of-the-art query engine. Also exciting to see Arroyo (YC W23) get a mention! The coolest thing about the paper (and about DataFusion itself) is that it demonstrates how building the engine in an open, extensible way doesn't preclude great performance. Systems built on top of DF (like Arroyo) can easily add new functions, operators, tree-rewrites, optimizations, data types, and more while using the the same tools and infrastructure that the built-in functions/operators/etc. are built with. This is incredibly powerful, and has already led to a blossoming of powerful data systems including Comet (a Spark accelerator), LanceDB (a vector DB and query engine), and ParadeDB (which embeds an OLAP DB in postgres). Read the paper here: https://lnkd.in/dpknuPy4