Adam Alpire

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Kommentare

Problem: There are thousands of language models (LMs) on 🤗 HuggingFace - but which one is the best for your NLP task? Solution: Use ⚖️ TransformerRanker ⚖️!

1

In Daniel Klitzke's latest article, he shows how you can use the Huggingface Transformers and Sentence Transformers libraries to boost you RAG pipelines using reranking models. #LLM #MachineLearning

23

Memorystore for Redis is a great platform for AI applications like Retrieval Augmented Generation (RAG) because of its very fast vector search capabilities. It can now store and search over billions of vectors with extremely low latency thanks to new support for Memorystore for Valkey and Memorystore for Redis Cluster. This is achieved by partitioning the vector index across multiple nodes. In addition to increased scale, Memorystore now supports "hybrid queries" which combine vector searches with filters on other data fields. This allows for more precise and relevant results, like filtering products by price and type in an online clothing store. Finally, Google is committed to open source and is working to donate these vector search capabilities to the Valkey key-value datastore. https://google.smh.re/4Nvl

2

💥💥💥 Reciprocal Learning Julian Rodemann · Christoph Jansen · Georg Schollmeyer Abstract We demonstrate that numerous machine learning algorithms are specific instances of one single paradigm: reciprocal learning. These instances range from active learning over multi-armed bandits to self-training. We show that all these algorithms not only learn parameters from data but also vice versa: They iteratively alter training data in a way that depends on the current model fit. We introduce reciprocal learning as a generalization of these algorithms using the language of decision theory. This allows us to study under what conditions they converge. The key is to guarantee that reciprocal learning contracts such that the Banach fixed-point theorem applies. In this way, we find that reciprocal learning converges at linear rates to an approximately optimal model under some assumptions on the loss function, if their predictions are probabilistic and the sample adaption is both non-greedy and either randomized or regularized. We interpret these findings and provide corollaries that relate them to active learning, self-training, and bandits 👉 https://lnkd.in/d53Hg6vt #machinelearning

2

How to fine-tune #LLMs on custom #datasets at Scale using Qwak and #CometML Alexandru Razvant https://lnkd.in/gP4aWM69

3

“After one year of reading and working daily with LLMs, here are the 𝘁𝗼𝗽 𝟭𝟮 𝗯𝗹𝗼𝗴𝘀 on 𝗟𝗟𝗠𝘀 & 𝗥𝗔𝗚 I 𝗿𝗲𝗮𝗱 𝘄𝗲𝗲𝗸𝗹𝘆 and 𝗿𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱 ↓”

8

1 Kommentar

Blogged: Interoperability between lakehouse table formats with Apache XTable (Incubating). Scenario: Dremio -> Apache Hudi ❌ Dremio -> Apache Iceberg ✅ There are 2 analytics teams in an organization. Team A uses Hudi to manage some of their most critical, low-latency data pipelines. Team B focuses on ad hoc analysis, BI, and reporting, using Dremio's compute engine with Iceberg. ❌ Challenge: Team B wants to do analytics using data from the 2 different datasets within the organization. To do so, Team B has to use the dataset generated by Team A (stored as a Hudi table) and combine it with their dataset (the Iceberg table) ✅ Solution: Interoperability between the formats with Apache XTable. Detailed reading link in comments. #dataengineering #softwareengineering

105

4 Kommentare

Interoperability in Lakehouse Table Formats. At LinkedIn's Big Data Conference, I was asked how I see Apache XTable (Incubating) being adopted, and how this open-source solution can be applied to different use cases with lakehouse table formats. Or will it just be a standalone TOOL? XTable started with the core idea around “interoperability”. That you should be able to write data in any format of your choice irrespective of whether it’s #ApacheIceberg, Apache Hudi or Delta Lake. And then you can bring any compute engine of your choice that works well with a particular format (performance, integration wise) & run analytics on top. Each of these formats shines in some specific use cases. And so depending on your use case & technical fit (in your data architecture), you should be free to use anything without being married to just one. However vendor query engines (warehouse, lake compute) usually have support for only one of these formats. And even if they do work with multiple formats, it is practically tough to build optimization capabilities for each of these formats. So, to summarize, XTable will have 2 major applications: ✅ Vendor-side ✅ Applied-side (customers using multiple formats) On vendor side, the product will leverage XTable underneath & allow users to seamlessly access data irrespective of the 3 open table formats. The recent Microsoft Fabric OneLake-Snowflake interoperability is a critical example that solidifies what I have been saying. This is what OneLake Shortcuts does. - OneLake will support both Iceberg & Delta through the XTable project - Data written by either Snowflake or Fabric will be available in both Iceberg & Delta formats via XTable translation in OneLake This abstraction at a user level, will allow disparate data sources to work together as "one single copy" irrespective of the table formats. #dataengineering #softwareengineering

62

1 Kommentar

Data lakehouse capabilities of Apache Doris: ❓ Do you know Apache Paimon? ✍ It is a data lake format that innovatively combines the advantages of the #datalake format and the #LSM structure to introduce real-time stream update capability into the data lake architecture. ❓How is this related to Apache Doris? ✍ Apache Doris provides native support for some of the latest features of Paimon: 1️⃣ Supports various types of Paimon Catalogs, including #HiveMetastore and #FileSystem. 2️⃣ Natively supports Primary Key Table Read Optimized introduced in Paimon 0.6. 3️⃣ Natively supports Primary Key Table Deletion Vector introduced in Paimon 0.8. ❓What does this mean? ✍ It means users can enjoy: 1️⃣ Real-time data ingestion into the data lake: Accelerated by Paimon's LSM-Tree model, data ingestion can be finished within minutes. Paimon also supports aggregation, deduplication, and partial column updates, enabling more flexible data flows. 2️⃣ High-performance data processing and analysis: Paimon provides Append Only Table, Read Optimized, and Deletion Vector. These technologies can be integrated with Doris' powerful query engine to achieve fast queries. 🚀 Comparing Apache Doris 2.1.5 and Trino 422, with Primary Key Table Read Optimized (#Paimon 0.8) enabled, on TPC-DS 1000 dataset: On average, Apache Doris is 3~5X the performance of #Trino.

31

8 Kommentare

The #medallionarchitecture gets some hype here and there. Still, IMO, it's a revamped architecture from the classical data warehouse layering `stage -> cleansing -> core -> mart` we have done since the inception of modeling data warehouses with simplified names and optimized for data lakes. What do you think? 📷 credits Jorrit Sandbrink ＃＃ Some discussion from Twitter/X: > Gold is the bronze layer of ML Yes, sure, the whole ML, with its vector databases on top, will end up building its own DWH layering as it also needs to cleanse data or aggregate into a feature store. Basically, ML/AI does MLOps, and data engineers do DataOps. Both end up in the same; it might be more efficient to profit from existing data architecture and add a layer or use one for ML/AI. --- > The theory I have been hearing is that stage+ nomenclature is for technical implementation by dev teams while medallion one is meant to be used as business-facing vocabulary [...] I agree that gold, silver, and bronze are easier to understand and explain to people, but I don't think there's a difference between business/dev; it's the same, just a different name. To me, the gold layer is 100% the data mart, right? Why replaying such a well-defined term? Gold is simpler to explain to someone new, but to existing people, a data mart is usually known across. --- > My understanding was > Bronze - Deduplicated and flattened copy of data from the source > Silver - Facts & Dimensions > Gold - Business Oriented (Marts) I'd say you won't do dedups in bronze, except the tools already support that; you just copy 1:1 to avoid any error in the pipeline when stressing production OLTP db. Fact & dim in silver is good, depending on the architecture. You can model your DWH with a star schema in core (silver) or keep the 3NF as Jorrit showcases. Usually, keeping 3NF is silver ends up with a more straightforward, less complex data pipeline. Yep, gold = data mart would also be what I understand. --- More on https://lnkd.in/e3UuiYs5.

257

5 Kommentare

💛 working with TMDL on Direct Lake I tested my personal new highlight feature of this month today with Marcus Wegener and Andrea Hupp - it works on all explicitly created Semantic Models - the "edit mode" has to no report - like in the Web-Service. But the DAX query view feels "more close" - all changes are synched to the published model. No saving needed. 🤯 but the real game changer is to export the Power BI file a project, then I can: - edit everything in TMDL code and sync the changes fast to the published model (but also beeing able to refuse the changes). Deployment never have been so easy 🤩 - as an empty report appears when I open the .pbip file, I am able to test my changes in real visuals. Even when I love the DAX query view... sometimes visuals show me more insights when I investigate some data issues 😉 👾 Doing the Video was also a chance to use the AI Dubbing one more time - it still feels a little strange, but I still like it and its faster doing the recording multiple times. I still sound different in real english tutorials I do from time to time 😅 Looking forward for more news from the European Microsoft Fabric Community Conference ..! 😃 #powerbi #fabric #update

18

2 Kommentare

sigmoid , amazing method for classification of numbers #AI #MachinLearning #NLP

1

Meta has released Llama 3.2, which includes both vision and text models optimized for edge and mobile use: ->> Vision Models (11B & 90B): Drop-in replacements for text models, outperforming closed models like Claude 3 Haiku in image understanding. Fine-tuning is available with tools like torchtune and torchchat. ->> Lightweight Text Models (1B & 3B): Designed for mobile and edge devices, they support 128K tokens and excel in summarization, rewriting, and instruction tasks. Optimized for Qualcomm, MediaTek, and Arm processors. ->> Llama Stack Distributions: Simplifies deployment across environments (cloud, on-prem, on-device), supporting retrieval-augmented generation (RAG) and tooling-enabled apps with integrated safety features. ->> Ecosystem Support: Collaborations with AWS, Dell, Databricks, and others for enterprise clients. On-device distribution via PyTorch ExecuTorch and single-node via Ollama. ->> Open Availability: Llama 3.2 models can be downloaded from llama.com, Hugging Face, and partner platforms like AWS, Google Cloud, and Microsoft Azure. Meta emphasizes openness and modifiability to foster innovation. https://lnkd.in/gxCmurx8

6

This week I'm considering the benefits of so-called PostModern Data tooling (inspired by Tomasz Tunguz's recent post). Why is it such a good model to start with OSS and then offer a paid version? Shoutout to Dagster Labs, MotherDuck, Tobiko, ClickHouse, Lightdash and of course Cube, who I count among these tools. https://lnkd.in/e-nj_XUS

13