Robert Thompson

My new article on medium: how to simply enable temporal / history table in PostgreSQL. No need for: - Special extensions. - Specialized databases. - Special techniques like event storming or soft deletes. None of that is needed. Don't overcomplicate things. https://lnkd.in/dQDNrCq8

20

🎉 The Snowflake Model Registry is now available for general use through the Snowpark ML (snowflake-ml-python) package version 1.5.0. 🚀 With the Snowflake Model Registry, you can securely store, manage, and use machine learning models in Snowflake, where your data is already stored. The registry supports most popular types of Python ML models and can be accessed from both Python and SQL. 💻rom both Python and SQL. The registry supports the following types of models: ⚫ Snowpark ML Modeling ⚫ scikit-learn ⚫ XGBoost ⚫ LightGBM ⚫ CatBoost ⚫ PyTorch ⚫ TensorFlow ⚫ MLFlow PyFunc ⚫ Sentence Transformer ⚫ Hugging Face pipeline ⚫ Other types of models via the snowflake.ml.model.CustomModel class https://lnkd.in/geGsmYUZ #snowflake #infostrux #datasuperhero #snowflake_advocate

10

Fascinating new paper from the legendary Mike Stonebraker about present and future of databases titled: “What Goes Around Comes Around…And Around…” MapReduce: MapReduce systems, of which Hadoop was the most visible and (for a time) most successful implementation, are dead. Key-value stores: These systems (Redis, RocksDB) have either “matured into RM [relational model] systems or are only used for specific problems.” Document stores: NoSQL databases that store data as JSON documents, such as MongoDB and Couchbase, benefited from developer excitement over a denormalized data structures, a lower-level API, and horizontal scalability at the cost of ACID transactions. However... (read it yourself) Columnar database: This family of NoSQL database (BigTable, Cassandra, HBase) is similar to document stores but with just one level of nesting, instead of an arbitrary number. However,... Text search engines: Search engines have been around for 70 years, and today’s search engines (such as Elasticsearch and Solr)continue to be popular. However ... etc, there's a lot more in the article https://lnkd.in/di7WZJcw

19

https://lnkd.in/gUpFEbby One of the limitations of current generation #LLM is it's inability to consume non linear data at scale. #SpreadSheetLLM Attempts to get rid of this limitation by focusing on most commonly available non linear data, that is tabular data.

6

1 Comment

We just launched two big updates in #AzureAISearch that will level up your retrieval systems: Query Rewriting (QR) and a Next-Gen Semantic Ranker (SR). These features are designed to make your RAG (Retrieval-Augmented Generation) apps smarter, faster, and more cost-effective. Here’s what’s exciting: 🔹 Semantic Ranker works out of the box to rerank the top 50 results from your search. It delivers up to +22 NDCG@3 improvement and runs 2.3x faster than before. 🔹 Query Rewriting transforms your queries (up to 10 rewrites!) to improve recall, making it useful for tricky queries like misspellings, keywords, or mixed formats. Together, QR + SR are a powerhouse duo that help you get the most out of your search setups. Whether you're using text, vector, or hybrid search, they bring a huge relevance boost—even for compressed vector indexes with binary quantization. In this blog, Alina Stoica Beck breaks down all the details, from benchmarks to real-world datasets, and shows how QR + SR can take your search stack to the next level. https://lnkd.in/ewhTtSut

171

16 Comments

Anthropic’s Claude 3 Opus foundation model, the most advanced and intelligent model in the Claude 3 Family, is now available on Amazon Bedrock. Claude 3 Opus is Anthropic’s most intelligent model, with best-in-market performance on highly complex tasks. It can navigate open-ended prompts and sight-unseen scenarios with superior fluency and human-like understanding. With Claude 3 Opus on Amazon Bedrock, you can execute complex actions, interactive coding, research review, hypothesis generation, drug discovery, advanced analysis of financials and market trends, and forecasting. Just like the other Claude 3 models, Claude 3 Opus has state-of-the-art vision capabilities that can process and analyze images. These models were trained to be extremely capable at understanding photos, charts, graphs, technical diagrams, and optical character recognition (OCR).

8

While researching on net, I stumbled upon this open source data visualization framework by Apache called Superset. Superset has got *60.1 K stars on Github*. Superset is a modern data exploration and data visualization platform. Superset can replace or augment proprietary business intelligence tools for many teams. Superset integrates well with a variety of data sources. Superset provides: - A no-code interface for building charts quickly - A powerful, web-based SQL Editor for advanced querying - A lightweight semantic layer for quickly defining custom dimensions and metrics - Out of the box support for nearly any SQL database or data engine - A wide array of beautiful visualizations to showcase your data, ranging from simple bar charts to geospatial visualizations - Lightweight, configurable caching layer to help ease database load - Highly extensible security roles and authentication options - An API for programmatic customization - A cloud-native architecture designed from the ground up for scale Learn more from here: https://lnkd.in/gDxeNe3m #data #DataExploration #DataVisualization #AI #Analytics

7

If you ingest data from Kafka/Kinesis into Databricks, you should watch Charles Tan and Shawn Gordon's short vide on how to shift left your data pipelines to reduce cost and latency! #ApacheFlink #Kafka #StreamProcessing #RealTimeData https://lnkd.in/ggMsi9KU

14

1 Comment

Large language models are wonderful at ingesting large amounts of content & summarizing. Uploading an academic paper and I can pester it with an infinite list of questions & it will respond with equally infinite patience. In comparing the two most recent Microsoft earnings calls, Claude highlighted: - faster than expected Azure growth (29% vs 27%) - AI contributing 8% of Azure revenue up from 7% - higher CapEx spending & greater capacity constraints for data centers - much better commercial bookings growth : 17% vs 13% Excellent analysis. Benn Stancil described LLMs as great averagers of information. But what if I don’t want the average? If I seek the data point two standard deviations out? If I’m channeling my inner Anthony Bourdain & I’m seeking the fermented shark or cobra heart within my query? At Google, we ranked web pages & ads with many signals. An engineer taught me that of them found rarity of a word across a set of documents. For example, if there’s a document containing 10 instances of the word ghotiacross a collection of documents, where the average document has 0 ghotis, that document is likely the best to answer a search about fish. I haven’t found a way to goad an LLM to produce the rare result. Why did one user decide to use a particular piece of software when 10 others did not? Maybe I haven’t learned how to prompt an LLM well. Ideogram launched a feature called Magic Prompt that expands a basic prompt into richer instructions that elicit better results, marrying the language of a user & a computer. A user might write : A boy with a dog in a park. Magic Prompt replies : A heartwarming scene of a young boy playing with his happy, wagging dog in a lush, green park. The boy has a playful smile on his face while he tosses a bright yellow ball for his furry friend to fetch. The park is filled with beautiful flowers and trees, creating a serene atmosphere for this bonding moment between boy and dog. Magic Prompt surprised me in three ways. First, unlike Google search where terse queries tend to work very well, LLMs relish verbosity. Less Hemingway & more Faulkner. Second, maybe there’s a prompt for rarity. If you know what it is, please share with the rarity shibboleth. Third, I wonder what other prompt techniques are out there that have yet to be discovered to wrest the best from an LLM. This technique is called TF-IDF. It means Term Frequency Inverse Document Frequency.

21

5 Comments

S3 Table Iceberg: The credential vending issue still exists. remove_orphan_files failed-- I need to give AWS creds. I had this issue in Snowflake too. All other maintenance procedures are working--Expirte, rewrite, etc. In snowflake, spark.sql(f""" CALL s3tablesbucket.system.remove_orphan_files(   table => 's3tablesbucket.prod.example_table' ) """).show(1000,1000) Py4JJavaError: An error occurred while calling o36.sql. : java.io.UncheckedIOException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3" How to fix this issue: In Spark Session, we need to provide AWS creds. --conf spark.hadoop.fs.s3a.access.key=<> \ --conf spark.hadoop.fs.s3a.secret.key=<>\ --conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem Roy Hasson Amazon Web Services (AWS)

18

6 Comments

In the early 2010s, every new data analytics system used Hadoop as the easy-to-beat baseline. It became so common that I've seen other domains referring to their weakest competitor as "the Hadoop" of that space. There were three reasons why. First, Hadoop was the first serious FOSS in its class, making it easy to compare against. Second, it had real-world users because of the barriers to building in-house alternatives. Third, its first-attempt design made it easy to improve upon. Flink seems to be "the Hadoop" of the streaming world. As the first mature FOSS solution, it opened doors for engineers to adopt stream processors. But its assembly-like programming model, lack of semantics, operational burden, and inefficient JVM-based design leave plenty of room for improvement. Spark emerged the clear victor over Hadoop for several reasons. I obviously have my own biases about what the analog in the streaming world would look like. :)

29

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Comments

Customers are using the Apache Iceberg format because they can read and write data concurrently using different frameworks. Iceberg format also allows for time travel and rollback so you can run SQL queries on a point-in-time snapshot of your data, or rollback data to a previously known good version. Now, you can stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose. https://lnkd.in/g8g3FUah #Data #StreamingData #DataStreaming #Firehose #S3 #AWS #ApacheIceberg #Iceberg

50

Metadata Indexing in Apache Hudi for faster performance. Data skipping is one of the common techniques used with large volumes of data to achieve better query performances. The whole idea is simple - read as less data as possible! What that means is that as a compute engine paired with a lakehouse platform like Apache Hudi it should read only the data files that are needed to satisfy the query from the storage. Data skipping not only reduces the volume of data that needs to be scanned & processed, but it can also lead to substantial improvements in execution time. Of course this is made possible because of the metadata information provided by file formats like #Parquet. Basically, each Parquet file contains the min/max values of each column along with other useful info such as: num of NULL values. These min/max values are 'column statistics'. Now, although, we could directly leverage these stats for data skipping, this could affect the query performance because the engine still has to go through each file to read the footer. ❌ This process can be very time taking specifically with large data volumes. What's Hudi's approach? ✅ Hudi adds a next level of pruning in this case. It basically takes all these column statistics & collates the info in the form of an INDEX. ✅ Indexes such as this (& more) are incorporated into Hudi's internal metadata table so engines can just get the files where the data is stored. Therefore, instead of reading individual Parquet footers, compute engines can directly go to the metadata table's index & fetch the required files. Way faster. For e.g. in this image, there are 2 parquet files that has well-defined value ranges for each of its columns. - File1.parquet contains 'Salary' ranges in $10000-40000 - File2.parquet contains 'Salary' ranges in $45000-90000 Now, if we run a query to fetch all the records where Salary=30000, the engine (Spark) can fetch the records from only File1 as it has values within the range & skip the other one. Imagine doing this at scale (with large number of files)! #dataengineering #softwareengineering

88

2 Comments

A great guide on optimization of Spark ETLs using the Spark UI, thanks Peter Stern!

1

AWS's announcement of S3 Tables and S3 Metadata isn't just another feature release - it's a strategic move in the data lakehouse space. Here's why this matters for CTOs & technology leaders: AWS's decision to build on Apache Iceberg is particularly interesting. Rather than pushing their own proprietary format or backing Apache Hudi (where they're the biggest contributor), they've aligned with what's becoming the de facto industry open table spec. ICEBERG ! This is a smart play that acknowledges a reality in modern data architecture: organizations want standards-based, interoperable solutions that don't lead to vendor lock-in and more of copy data syndrome. The performance claims are compelling - 3x faster queries and 10x more transactions per second. But what's more interesting is the automated maintenance features. The potential for 30-60% cost savings through automatic tuning isn't just about economics; it's about engineering efficiency. Your teams spend less time managing infrastructure and more time delivering value. Yes, you'll pay 15% more for storage and new fees for monitoring ($0.025/1000 objects) and compaction ($0.05/GiB). But let's be honest - what's more expensive: slightly higher storage costs or engineering time spent managing data optimization? For most organizations, the automated management will likely pay for itself in reduced engineering overhead. The hidden gem of S3 Metadata feature might seem less exciting, but it's solving a critical problem. Many organizations have built homegrown solutions to track what's in their data lakes. Having this capability built into S3 could eliminate a whole category of custom tooling that engineering teams have had to maintain. What Engineering Leaders Should Consider?? Adoption Strategy: If you're already using Iceberg, the transition path looks straightforward. If you're heavily invested in other table formats, you'll need to weigh the benefits against migration costs. Team Skills: While this simplifies many operations, it's still Apache Iceberg under the hood. Your team will need to understand Iceberg concepts to fully leverage these features. Cost Models: The pricing model rewards efficient data organization. It's worth reviewing your current data patterns to estimate the real cost impact. The integration with IAM for table-level RBAC, automated maintenance, and metadata management suggests a future where data lakes can be both powerful and manageable. For technical leaders, the key question isn't whether to use these features, but how to integrate them into your data strategy. The potential for reduced operational overhead and improved data governance makes this a compelling option, especially for organizations already heavily invested in AWS. One thing's certain: Iceberg won the open table format war, but the data lakehouse wars just got a lot more interesting. *Views are mine. #datastrategy #dataarchitecture #dataengineering #lakehouse #modernization #ai

126

11 Comments

I share because the meta-message of making sure you've done the appropriate level of diligence before picking your database (or any architecturally significant component) is a much-needed message IMHO. I've seen too many examples in the past few years of decisions made without the appropriate level of diligence and any regard for the unnecessary costs these decisions impose on the business. https://lnkd.in/eD_24953

4

Really looking forward to the legal oversight clarifications around how AI agents execute work. Shubham’s is a really fascinating example of 5 agents with one conductor working in tandem together. Truly, AI history we’re watching play out. Good, bad, neutral it’s historic. #LearnAI #AIagentd #AIeducation #AIethics #AI

2

It looks like that the new, upcoming PostgreSQL 17 is mostly performance oriented. Many improvements to the optimizer. Here are the release docs, see for yourself, maybe I'm wrong. https://lnkd.in/dyUw4d9w

7