Nick Reich

108 Recipies Snowflake Build: Just moments ago (Nov 12th, 12:30 PM CST), SF presented the Snowflake Dynamic Iceberg table using Polaris External CatalogSync." We have 20+ recipes in this e-book. Below is how to build summary tables as new data streaming into the base table. Snowflake

6

How does an organization benefit from Strategic Data Modeling? 💥 Consensus and visibility of the broader business model 💥More productive conversations with business teams 💥Better quality of requirements 💥Higher signal, lower noise in technical conversations 💥Cross-platform, cross-domain, and widely understood conventions 💥Big-picture visual overview of the business and its database footprint 💥Preliminary designs become implementation blueprints 💥Accelerating onboarding of new team members 💥Making data more accessible and unlocking self-service within organizations 💥Keeping the database landscape manageable at scale 💥Getting a handle on complex data pipelines Check out Serge Gershkovich's book, 𝐷𝑎𝑡𝑎 𝑀𝑜𝑑𝑒𝑙𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑆𝑛𝑜𝑤𝑓𝑙𝑎𝑘𝑒, to learn more! #DataModeling #SqlDBM #Cloud #Snowflake https://lnkd.in/gsQsJ_yb

14

1 Comment

it is getting more convincing to embrace the philosophy of bringing AI to data...

2

Check out both the talks from #posetteconf about pgvector + Python! One from me about pgvector with everything but Django, then one from Paolo Melchiorre on Django: https://lnkd.in/g6ixPn_8 #python #postgresql #pgvector

11

LakeFlow + Delta Live Tables + Serverless Autoscaling Compute sound like a perfect solution to load all required data with data expectations / monitoring / alerting to the Lakehouse :)

12

I really enjoyed my conversation with Bryan Wood and Jeremy Kuhnash from RedPanda at the Databricks conference! I remember bumping into Redpanda when I was researching and writing my ebook for Redis (Redis Streams vs. Kafka - a visual guide) nearly 2 years ago! https://lnkd.in/ga4vbvaN For those who don't know, RedPanda is a faster, a more cost-effective, and a drop-in replacement to Kafka. For example: say for some workloads, you may need 10 Kafka brokers, with RedPanda, you'd need just 3 - that's a huge savings! So if you are thinking of using Kafka or Confluent, you should definitely check them out! https://redpanda.com #kafka #confluent #streaming Databricks

35

The first post in my “Understanding Apache Iceberg’s Consistency Model” is out. This post covers the internals of the read and write path, details on metadata manipulation, concurrency control, and so on. If you ever wanted a more in-depth post about Iceberg internals, this is for you. https://lnkd.in/dc8K8e2V Parts 2 and 3 coming soon, which dive deeper into the consistency model and formal verification work.

128

2 Comments

https://lnkd.in/eHbF9p5Y Vector Search has the unique ability to sync up a Delta table with a Vector Search index. Good-bye to much of time-consuming embeddings generation pipeline.

14

A common Databricks cluster misunderstanding is people think Spot instances will get them huge cost savings instantly... the reality is a bit more complicated... When Spot nodes are revoked randomly in a Spark cluster, the impact on the job's runtime can vary wildly. We've seen runtimes jump up 2-3X, even with just 1 worker revoked. This can cause the total cost of a job to even surpass "on-demand" prices! What can you do about this? Unfortunately Spot is really a random chaos machine and nobody has control over it (not even Databricks). Hopefully on average your costs are lower, but a static on-demand and optimized cluster can get you much more stable and possibly cheaper jobs. Be kind to your clusters folks! Sync Computing #dataengineering #AI

34

4 Comments

On the surface building a RAG system for your data applications may seem simple: "Vector DB + LLM = Done!" But under the hood things can get really complicated really fast. Great article on some of the pitfalls of homegrown RAG. TLDR: Costs are way higher than you think - Infrastructure costs, Personnel costs, Operational costs. Maintenance Horror Show - Latency, poor response quality, data processing issues, API quotas and infrastructure issues. Pre-processing complexity - Garbage In = Garbage Out. Without properly ingesting different data sources everything falls apart. Time-to-Market Reality - Your competitors are deploying production solutions while your business is losing opportunities.

10

2 Comments

I'm personally really excited about the newest Public Preview from Databricks and can't wait to test it out. With Databricks Apps, you can deploy custom Python applications directly within the Databricks environment on top of your Lakehouse data. No need for a separate web server to manage and host your application. In my previous consulting roles I spent a lot of time and effort making the results of ML models, custom analytics, and data engineering pipelines accessible to non-technical users and this feature would have been a true game-changer. Some sample applications that I'm excited to use Databricks Apps for: * Publish a custom, branded chatbot trained on your enterprise's data * Make results of ML models accessible to business users without the need to log into Databricks * Create interactive visualizations on top of your Lakehouse * Abstract complex query language with an easy to use interface

2

Given my experience with LangChain, building PDF document chunks and generating embeddings through the Palantir Build Pipeline is straightforward. Before you learn this tool, learn the following: Chunks, Embeddings, VectorDB, Similarity Search, Ranking, Prompt, etc in LangChain. Of course, Pyhton is your AK47 Palantir Technologies

10

Our recent discussion with Tobias Macey on The Data Engineering Podcast Data Quality, Observability and DataOps -- when will they be mainstream? https://hubs.ly/Q02KtXLn0 #dataobservabiliity #dataquiality #dataops

1

1 Comment

Snowflake should at least tell us what bugs they are fixing and what features they are working on in snowflake managed Polaris. It will helps us to prepare .. For example: add_files, register_table procedure are not working. I know snowflake team is working on it but no info to community. Some of these features essential to move on from PoC to prod. James Malone Snowflake

8

An early morning riff with Claude on the issues I've experienced over the year working on Snowflake/Databricks/dbt specific data infrastructure. Anyone else feeling this kind of pain? “I have a multi-account Snowflake set up (think 9 different regions), and I need to ensure that all my non-service users have MFA enabled - how can I do this without manually logging into each account? And how can I ensure that any new user that gets created also adheres to this policy?” “I have a very complex Snowflake set up, with multiple Snowpipes. The only real testing that we currently have for whether the Snowpipes are working properly is dbt tests, which are way further downstream. Worse, when there’s a failure, I have to analyze the dbt logs to understand what happened. How can we be more informed about Snowpipe status? Or more generally, any Snowflake resource?” “I’m trying to implement Terraform/Titan/Permifrost (or basically any IaC tool) on a fairly complex Snowflake setup. There has to be a way to create an “image” of my Snowflake setup, and have that automagically transformed into Terraform code?” “Dude I’m an analytics engineer and I don’t really wanna Terraform” “I’m getting started on Snowflake (or Databricks) and I want to quickly learn about best practices when it comes to security, roles, etc. Is there a reference architecture (beyond a 4 year old dbt blog post about LOADER/TRANSFORMER/REPORTER roles)” “What if we could analyze a dbt PR and assess the impact on 1) security and 2) query performance in a CI/CD process?” cont'd in comments below

3

7 Comments