Colin Riddell

How does an organization benefit from Strategic Data Modeling? 💥 Consensus and visibility of the broader business model 💥More productive conversations with business teams 💥Better quality of requirements 💥Higher signal, lower noise in technical conversations 💥Cross-platform, cross-domain, and widely understood conventions 💥Big-picture visual overview of the business and its database footprint 💥Preliminary designs become implementation blueprints 💥Accelerating onboarding of new team members 💥Making data more accessible and unlocking self-service within organizations 💥Keeping the database landscape manageable at scale 💥Getting a handle on complex data pipelines Check out Serge Gershkovich's book, 𝐷𝑎𝑡𝑎 𝑀𝑜𝑑𝑒𝑙𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑆𝑛𝑜𝑤𝑓𝑙𝑎𝑘𝑒, to learn more! #DataModeling #SqlDBM #Cloud #Snowflake https://lnkd.in/gsQsJ_yb

14

1 Comment

🔗 Attention Prime Vendors: Understanding C2C Positions Prime vendors, we understand the reservations you may have regarding Corp-to-Corp (C2C) positions. Here’s a breakdown of why C2C arrangements are typically avoided, and how I can be convinced to consider me. Reasons I Hope for Avoidance: Risk Management: Mitigating legal and financial risks is paramount, and direct engagements often seem safer. Contractual Complexity: C2C arrangements introduce additional complexities, from tax implications to compliance issues, potentially complicating operations. Client Relationships: Maintaining strong client relationships is essential, and C2C arrangements may complicate these ties, particularly when clients have specific preferences. Control and Oversight: Direct engagement provides greater control and oversight over contractors, simplifying management. Convincing Factors: Benefits: With C2C arrangements I offer cost savings, flexibility, specialized expertise, and reduced administrative burdens, benefiting both parties. References: I have successful C2C engagement with a well reputed consulting company called Signature Consultants which is to illustrate reliability, professionalism, and capability. Negotiate Terms: Flexible with the pay rate per hour. Understanding the landscape and benefits of C2C arrangements can foster collaboration and open doors to mutually beneficial partnerships. Let’s explore the potential together. #C2C #VendorRelations #CollaborationOpportunities

1

It is a tough job to maintain a large shared Dataproc cluster. You’re bombarded with questions from your team and other application teams about Spark job performance. “Why is my job slow?” “Why hasn’t it started yet?” “Did we mess up the cluster size?” “What’s going on with autoscaling?” “Is one job hogging all the resources?” I built a custom Spark dashboard to shed some light on these mysteries. It’s designed specifically for Dataproc on GCE. #dataproc #apachespark #googlecloud #googlecloudconsulting

18

Snowflake Cortex + Streamlit + dbt Semantic Layer = Data Querying interface in Natural Language

8

1 Comment

It was great talking with The Register about the early history of Apache Spark and some of its current use cases. Lots more under way in Spark 4.0! https://lnkd.in/gAgVdasw

438

3 Comments

Check out the upcoming webinar on getting started with SDF -- the product built at my company!

7

Well said James, This is one of the most common asks from customers - can Horizon features be available for Polaris tables? Check out this demo video to see how Polaris tables can be visible in Horizon and how all Horizon governance functionalities can be used on Polaris tables.

15

3 Comments

Tired of constant prompt tweaking for LMs? Meet DSPy: a framework that makes building reliable LM systems more like programming, less like guesswork. DSPy optimizes LM pipelines effortlessly with declarative programming that adapts to your data and models. With DSPy, you can: - Create workflows with reusable modules. - Use optimizers to auto-tune prompts and parameters. - Focus on system goals, not prompt hacking. Think of DSPy as the PyTorch for LMs: structure and optimization for your tasks. Let's make LM engineering smarter — less prompting, more programming! https://lnkd.in/g4wEfSCq Keishin Nishiyama Iris Fu Adam King John Hawkins

7

Snowflake Dynamic Iceberg Summary Tables: Show me the meat! I’ve tested the Dynamice Iceberg Summary table and added recipes to my e-book. However, I haven’t yet tested performance. My goal is to ingest 20 million rows (15 columns) every 5 minutes using Kafka streaming and to build 4 KPIs with a dynamic Iceberg table. This will result in a table size of approximately 850 GB per day and around 9,500 files. We plan to retain this data for four months. What is my cost?

8

2 Comments

We've got a 🆕 use case for data engineers that shows how to integrate Soda checks into your Dagster Labs-powered ETL pipelines, making it easier than ever to maintain data quality throughout your workflows. In this use case, 🚲 Bikes 4 All is an organization that runs bicycle retail stores across multiple regions. The Data Analysts have been struggling with sales forecasts and 📈reporting dashboards. To fix this, the Data Engineering team has automated the ⛓ ETL pipeline using Dagster and dbt Labs, orchestrating the ingestion and transformation of data before exporting it for business intelligence analysis. 👀 Check out how Soda's data quality checks are embedded at multiple points in the pipeline, ensuring that data transformations are accurate and reliable before being passed downstream for analysis. 🎯 By automating these checks, the Bikes 4 All team ensures that their sales forecasts and reports are built on trustworthy, high-quality data, empowering their analysts with accurate insights for the right business decisions. 📕 Here's the guide: https://lnkd.in/ea4n-tvd Good luck, happy testing! #dataquality #dataengineering #datatesting

26

1 Comment

Rewrite bigdata in Rust!

8

1 Comment

Got to spend a few minutes using Sigma Computing leveraging a connection to Databricks. Here is a quick video of me easily putting together a dashboard. All in all, took about 10 minutes to build what you see here from scratch, including setting up a connection to Databricks (not shown in video). If you'd like to read my take on Sigma, the pros and cons, consider checking out my blog post I did for SunnyData @ https://lnkd.in/e9v7PKqp. Also consider checking Sigma out yourself!

50

2 Comments

Snowflake Polaris Catalog: Now table-level privileges can be granted. (We are working on to update 108 recipes doc. ) ✅ Object-level privileges: Apache Polaris supports scoping privileges at the catalog, namespace or table levels. This is now available in the Snowflake Open Catalog interface. ✅ Table details: You can now view table schema and privileges in the Snowflake Open Catalog user interface. Snowflake Dremio

4

After many public and private requests, I will be doing a series of hands-on takes on Fabric vs Databricks. Will Fabric impress me as much as Databricks? Will it perhaps win or lose by a thread? Stay tuned over the next few weeks! I will say this: I will be extremely critical in my comparisons/reviews, no sugar coating anything. Want to help? If you are a Fabric expert, I would love the opportunity to pick your brain. If you are part of the Fabric team, I'd love the opportunity to ask you questions along the way as well to faithfully and fairly represent my research. Stay tuned! SunnyData

112

26 Comments

Reduce your Report creation time with copilot for DAX management.

2

Recently, I faced an issue where one of my critical ETL pipelines, which uses Apache Hudi, was blocked due to corrupt metadata Cleaner task files. After several unsuccessful restarts, I had three recovery options: 1. disable / enable Hudi metadata 2. restore from a daily backup (which would be slow but effective) 3. manually remove the problematic Cleaner task files. I opted for the third option which allowed me to recover the ETL quickly. However, I remain cautious about using Hudi’s metadata features due to these reasons: limited query engine support, stability issues like broken Bloom Indexing in version 0.14.1, and the potential for data integrity problems if the metadata becomes corrupted. #ApacheHudi #ApacheHudiMetadata https://lnkd.in/gzQS_Yrk

7

2 Comments

Alongside a handful of folks with solid content and community contributions, I've been recognized as a Databricks MVP (formerly Beacons). Here are some thoughts around it: 1) My content hasn't been all "hip-hip, hooray". Historically, about 30% of my Databricks content has included constructive criticism, sometimes of key strategic Databricks offerings. Still, I was asked to be part of the program. Key takeaway: Authenticity matters, specially when paired up with a healthy product culture. 2) I've loved working with Olaf Hubel and Torey (Markowitz) Bublitz, and the entire DevRel team from Databricks. I remember in particular talking with Olaf at the Databricks Data + AI Summit and him just encouraging me to keep it real with the content I put out. 3) I still don't like the term influencer, so I am thankful the program is not called "Databricks Influencers". 4) Lots of solid content writers in the group. The ones that I personally always stop to read are from 👨🏻‍💻 Jaco van Gelder, Maria Vechtomova, Hubert Dudek, and Vishal Waghmode. They all cover some areas or angles that I don't usually, cover, so useful for gaining perspective. Notable mention to Youssef Mrini, who is a Databricks employee and not a Databricks MVP, but does put out a lot of great content. 5) Don't look up to us and take what we say as always being right. Test things out yourself.

356

47 Comments

Data folks, (in particular dbt users)... don't you just hate it when a dbt pipeline fails (a test or otherwise) leaving broken data for everyone to see and (unknowingly) use... What if you could configure dbt to roll back the tables to their previous states automatically on failure? Well turns out you can (in bigquery): https://lnkd.in/dgipWibU

29

3 Comments

For folks who use BigQuery, did you know that there are three ways to insert data? A slow way, a fast way, and a very very fast way. All come with trade-offs, and in my blog post, I talk about how to use them. I also give away a TON of code that would have saved me about a week of time. Check it out! https://lnkd.in/gna8yHnY

14