Alex N.

Are you looking to work with Delta Lake but want to avoid the cost and complexity of Databricks? The Rust-based "delta-rs" crate provides a compelling alternative, allowing you to create and append to Delta Lake tables without the need for a Spark cluster. Some key benefits of the delta-rs approach: **Cost Savings**: Run Delta Lake operations in cheaper, more scalable environments like AWS Lambda. **Operational Simplicity**: Abstracts away the complexity of Spark cluster management. -**Flexibility**: Use delta-rs in diverse contexts, like Kafka message ingestion. Learn more about how you can implement Delta Lake without Databricks in our latest technical blog post: https://lnkd.in/gncb_Dkj #DataEngineering #DataLakes #DeltaLake #Rust

8

🤖 Should you use Databricks for ML? We take a look in our new Medium blog where we created a proof of concept integrating Databricks with GoodData's semantic layer for machine learning. See what we learned: https://hubs.ly/Q02PQ0f40

3

If you're new to Databricks or are looking to further develop your skillset, the Databricks academy is a great place to start. Here you can learn about Databricks sales and technical training to get enabled on the Data Intelligence Platform. Before you can help your customers unlock the potential of their data, you need to master the Databricks Lakehouse Platform. The Databricks Academy is your go-to place for training and certification for all roles in your organisation. If you would like to discuss the resources available and where to begin, feel free to reach out and I can point you in the right direction. #learning #certified #databricks

23

🚀Real-world SQL agent from Waii Real-world SQL is messy and complicated. If you want to build an agent over SQL, it needs to be pretty complex and detailed LangGraph is great for tasks like this, and this blog from Waii shows how to do so Blog: https://lnkd.in/gJRYGwDB

119

5 Kommentare

My Substack post, “The Data Engineer is Dead, Long Live the (Data) Platform Engineer”, has just reached 100 likes—a first for me! What’s even more interesting is that this post was written 1.5 years ago, but it has recently gained a surge in both reads and likes. Maybe it reflects a growing trend in the data space, or perhaps Substack’s recommendation algorithm decided to shine a light on it. Either way, I’m thrilled to see the conversation around the evolution of data roles and platform engineering continue to spark interest. The post dives into the shifting responsibilities in data engineering and why the role of a platform engineer is becoming increasingly vital in enabling scalable, reliable, and self-serve data platforms. If you’re interested in these topics, feel free to check it out here: https://lnkd.in/d4hamBA4 Big thanks to everyone who’s read, liked, commented or shared the post—it means a lot to see these ideas resonate with the community! Have you seen similar trends in your own work with data teams? I’d love to hear your thoughts! #dataplatform #dataengineer

103

6 Kommentare

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Kommentare

My TL;DR of database choices in 2024: - Pick Sqlite iff you don’t need the db to be accessed over network - Pick Postgres otherwise - Never pick NoSQL (and I did my PhD on NoSQL) Shout out to this awesome article I read this morning for more depth: https://lnkd.in/dwriene8 Happy Sunday! (Ignore this post and don't think about databases today 😂)

902

114 Kommentare

Breaking down Apache XTable's Architecture. Apache XTable (Incubating) is an omni-directional translation layer on top of open table formats such as Apache Hudi, Apache Iceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Blog in comments for a detailed read. #dataengineering #softwareengineering

73

3 Kommentare

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Kommentare

What questions do you have about Databricks Unity Catalog? Send us a DM or leave a comment below! We're hosting an #AMA tomorrow, 10/2 at 11 AM PT. Our Databricks Champion Shannon Lowder will be sure to answer your questions then. #UC #UnityCatalog #AskMeAnything

6

Apache Cassandra 5.0 is now General Availability (GA)! 🎉 This major release significantly improves performance, usability, and capabilities for the world's most powerful distributed database. Key features include: - Storage Attached Indexes (SAI) for improved query flexibility - Trie Memtables and SSTables for enhanced efficiency - JDK 17 support for better performance - Unified Compaction Strategy for improved node density - Vector Search capabilities for AI applications - Dynamic Data Masking for improved security 🔗 and more at: https://lnkd.in/e3k35MTK ⚠ As a reminder, the Cassandra 5.0 launch marks the end of the 3.x series. Users are encouraged to plan their upgrade strategy soon. 🔗 Blog: https://lnkd.in/eJT-VsuF ⬇ Download: https://lnkd.in/dU__DRmQ 🤝 Learn more about Cassandra 5.0 at CommunityOverCode in Denver, October 7-10, 2024. https://lnkd.in/gpY6fUkN

143

1 Kommentar

The new version of #apachepinot 1.2 is out now, packed with exciting enhancements! Check out the release notes here: https://lnkd.in/eNNfaAqc A recent blog post from StarTree delves into some of the standout features. Read more about the innovation in Apache Pinot 1.2 here: https://lnkd.in/e5wHtKgy

27

When you talk with me about data content, you will hear me complain a lot about the lack of analyst content. Usually, people show how to set things up but not how to work with the data. But then there is Parker Rogers and paradime.io - Parker created the first cool NBA analysis videos. With Paradime, they launched the first dbt & analyst competition with amazing results (I loved to go through the results). And what is even better: A new competition is coming up, and I am honored to be one of the judges. It's called: ➡️ dbt™ Data Modeling Challenge - Movie Edition 🍿 And truth be told, I like Basketball, but movies, this is something totally different. In a nutshell, here is how the competition works: From now until May 26th, data professionals from around the globe will use real movie and TV series data to craft SQL queries, develop dbt™ models, and derive insights, all for a chance to win a $1,500 Amazon gift card! Second and third-place prizes are $1,000 and $500, respectively! If you’re interested in joining or want to learn more, check out the link below! https://lnkd.in/dsdnhr8a

27

2 Kommentare

Bloom indexes in #PostgreSQL efficiently test for the presence of a set of values, especially useful for multi-column queries where traditional indexes may be less efficient. They are ideal for checking combinations across columns without needing explicit indexes on each. #PostgreSQLPerformance @MinervaDB #dba https://zurl.co/OKDl

5

You can’t edit or delete data in an Apache Kafka topic, but you can still fix bad data from causing problems in your event streams. Read Adam Bellemare's latest article on The New Stack for actionable advice on how to fix this problem.👇🏼 https://lnkd.in/eN4x6Z3A

20

Interoperability in Lakehouse Table Formats. At LinkedIn's Big Data Conference, I was asked how I see Apache XTable (Incubating) being adopted, and how this open-source solution can be applied to different use cases with lakehouse table formats. Or will it just be a standalone TOOL? XTable started with the core idea around “interoperability”. That you should be able to write data in any format of your choice irrespective of whether it’s #ApacheIceberg, Apache Hudi or Delta Lake. And then you can bring any compute engine of your choice that works well with a particular format (performance, integration wise) & run analytics on top. Each of these formats shines in some specific use cases. And so depending on your use case & technical fit (in your data architecture), you should be free to use anything without being married to just one. However vendor query engines (warehouse, lake compute) usually have support for only one of these formats. And even if they do work with multiple formats, it is practically tough to build optimization capabilities for each of these formats. So, to summarize, XTable will have 2 major applications: ✅ Vendor-side ✅ Applied-side (customers using multiple formats) On vendor side, the product will leverage XTable underneath & allow users to seamlessly access data irrespective of the 3 open table formats. The recent Microsoft Fabric OneLake-Snowflake interoperability is a critical example that solidifies what I have been saying. This is what OneLake Shortcuts does. - OneLake will support both Iceberg & Delta through the XTable project - Data written by either Snowflake or Fabric will be available in both Iceberg & Delta formats via XTable translation in OneLake This abstraction at a user level, will allow disparate data sources to work together as "one single copy" irrespective of the table formats. #dataengineering #softwareengineering

62

1 Kommentar