Matt Doherty

🏢 Quick take on Claude's new browser-based analysis tool for CRE folks Good for: ✅ Clean rent rolls ✅ Basic occupancy tracking ✅ Quick market comps ✅ Simple return calcs ✅ Bi-weekly payroll Skip for: ❌ DCF models (keep Excel) ❌ Yardi data dumps ❌ Complex waterfalls ❌ Portfolio rollups Real talk: Perfect for those "need it now" analyses in acquisition meetings, but won't replace your modeling tools. Think of it as your quick analysis sidekick. #CRE #PropTech #RealEstate

2

🚀 ETL vs ELT?💡  "The cost-effectiveness of ELT in the cloud era." What better way to spend a rainy Melbourne weekend than diving into a hands-on ELT pipeline project? Over the weekend, I built an entire ELT pipeline from scratch and experienced firsthand how learning by doing is the ultimate way to grow in this field. 📈✨. Here’s what I accomplished: -- ELT Pipeline Building with dbt, Snowflake, and Airflow -- 1️⃣ Set up a Snowflake account for data warehousing and explored its capabilities. 2️⃣ Worked with dbt Labs Core to transform data locally. 3️⃣ Learned building data models (including fact and dimension tables) using dbt's features such as sources, staging areas, and macros. 4️⃣ Explored Snowflake RBAC (Role-Based Access Control) concepts. writing both generic and singular tests for data quality 5️⃣ Used Apache Airflow for orchestration, deploying and managing my dbt project seamlessly. Using TPC-H sample data provided by Snowflake, I recreated a complete pipeline and learned the industry-standard tools step by step. At every stage, I realized the importance of tackling challenges hands-on rather than just reading technical manuals. Yes, there were moments when I felt stuck, but those moments made the process even more rewarding! 💪 🎉 Shoutout to Jay Zern Ng ! Thank you to Jay for his incredibly informative Youtube channel "jayzern", which guided me through this project. His videos not only explain the tools and processes in-depth but also showcase the importance of critical thinking and overcoming challenges while building an ELT pipeline. Highly recommend his channel to anyone diving into Data Engineering! Checkout this video: https://lnkd.in/g8SdUPp3 🔑 Key takeaways: - Learning by doing fosters creativity and problem-solving. - Persistence is key when navigating complex setups! 🌟 I’m excited to continue exploring the world of Data Engineering and honing my skills to stay competitive in today’s dynamic market. What about you? Have you worked on an ELT pipeline or explored these tools? Let’s connect and exchange ideas! 💬 #DataEngineering #ETLvsELT #CloudTechnologies #Snowflake #dbt #Airflow #LearningByDoing #DataPipeline #CareerGrowth #HandsOnExperience #DataAnalytics

14

Excited to try this one out! I'll be speaking at Minnesota Technology Association Tech Connect in a few weeks on Open Source Large Language Models and will discuss this and many more. Take read into why Snowflake's Arctic should be a model you explore with Streamlit. I'll give you a hint. "Arctic is on-par or better than both Llama 3 8B and Llama 2 70B on all metrics, while using less than ½ of the training compute budget." Look for covering in the upcoming Applied Artificial Intelligence Weekly News issue! https://lnkd.in/gM-wECeu

16

Did you miss this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #databricks

7

1 Comment

🐬 Dremio users deal with a lot of different data file and table formats. ⚒️ So Dipankar Mazumdar, M.Sc 🥑 of Onehouse and Alex Merced of Dremio worked together to create a new blog post. 🕵️ In the post, they show how you might use Dremio to smoothly do lakehouse analytics on Apache Hudi and Apache Iceberg data using Apache XTable (Incubating). 0️⃣1️⃣ Check it out - and borrow and revise the code as needed. https://lnkd.in/g-jBvJ55 #apachehudi #XTable #datalakehouse #dataengineering #onehouse #opensource #nolockin

28

3 Comments

How does an organization benefit from Strategic Data Modeling? 💥 Consensus and visibility of the broader business model 💥More productive conversations with business teams 💥Better quality of requirements 💥Higher signal, lower noise in technical conversations 💥Cross-platform, cross-domain, and widely understood conventions 💥Big-picture visual overview of the business and its database footprint 💥Preliminary designs become implementation blueprints 💥Accelerating onboarding of new team members 💥Making data more accessible and unlocking self-service within organizations 💥Keeping the database landscape manageable at scale 💥Getting a handle on complex data pipelines Check out Serge Gershkovich's book, 𝐷𝑎𝑡𝑎 𝑀𝑜𝑑𝑒𝑙𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑆𝑛𝑜𝑤𝑓𝑙𝑎𝑘𝑒, to learn more! #DataModeling #SqlDBM #Cloud #Snowflake https://lnkd.in/gsQsJ_yb

14

1 Comment

Why are iceberg data tables so hot right now ? - the TLDR before it melts ‘ -#AI needs structured #data of multiple variables to make a personalised offer or recommendation ( Netflix) Fast queries on vast amounts of data without sacrificing atomicity and granularity is key ( what worked for Netflix became Iceberg) .. Data ferreting by engineers and pipelines is a slow moving beast ( where the non Netflix world is ) … Everyone wants to be the Netflix of their industry using AI for personalization with Modern data infra that’s easy to use . #snowflake #databricks #confluent https://lnkd.in/g5V8S6gA

5

The classic data lake architecture limits to data silos, poor query performance, and limited real-time capabilities. We just published a new blog on #datalakes On Medium 🚀 "Navigating the Power of Real-Time Analytics with Apica InstaStore" Head over here to learn more: https://lnkd.in/g7izubRN #blog #data #cloud #apica #Medium #instastore #articles #datamanagement #observability

13

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

The recent announcement from Snowflake about the launch of their Enterprise LLM is hardly unexpected within the industry. What's unfolding before us is undeniably one of the most accelerated technological revolutions we've ever seen. The speed at which LLMs are advancing is truly remarkable, paralleled by their extensive and rapid adoption across enterprises, including IT and developer teams. At Aim Security, we're convinced that security leaders are currently presented with a rare opportunity to become the enablers of the GenAI revolution within businesses. Reach out to us today to discover how LLMs are deployed in your environment, and establish security guardrails to enable secure use. https://lnkd.in/duv53HVM

30

Why Snowflake is More Than Just a Data Warehouse When I joined Snowflake a couple of months ago, I saw it as a highly optimized SQL database, with an exciting side project in innovative areas like large language models (think Arctic) But, I quickly discovered that Snowflake is much more. It is a one-stop data platform that treats data as a first-class citizen—as we should! : ) 🌐 Unified Data Handling: Snowflake scales with you, handling structured, unstructured, and semi-structured data using SQL, Scala, Python, or Java. 🤖Top AI Research Institution: Introduced Snowflake Arctic, a top-tier enterprise-focused LLM that pushes the frontiers of cost-effective training and openness. Streamlit is also part of Snowflake now! 🛠 OCI-Compliant Container Services: Snowflake ensures your applications are ready to scale with OCI-compliant container services. 🐼 Run your Pandas code in a distributed manner by just changing the import statements—leveraging Python’s powerful data analysis library in the cloud. 🛒 Data Marketplace: Empower data buying and selling with just one click through Snowflake's Data Marketplace. ⚙️ GPU Support: Snowflake's GPU support provides the necessary compute power for training models or running complex inference. 💻 SQL-Powered AI: Run AI tasks using SQL commands, unlocking new possibilities with Snowflake. Run models like Llama, Snowflake Arctic family and Mistral AI’s mistral family and others as simple as : ) SELECT SNOWFLAKE.CORTEX.SUMMARIZE(review_content) FROM reviews; 🤖 GenAI Capabilities: Snowflake seamlessly integrates advanced AI and machine learning into workflows with its GenAI capabilities. 📊 Real-Time Analytics: Snowflake's architecture offers instant, actionable analytics, revolutionizing real-time insights. 🤝 Seamless Collaboration: Collaborate seamlessly with Snowflake's shared data architecture, ensuring sync with your team without lag or redundancy. 🔒 Security and Compliance: Snowflake prioritizes security with built-in compliance and governance features, ensuring data remains secure. Excited to be part of this journey! #DataScience #GenAI #Cortex #MachineLearning #AI #CloudComputing #SQL #Snowflake #Innovation #DataAnalytics #DataSecurity #Pandas #LLMs #SnowflakeCortexAI #ArcticLLM #EnterpriseAI

23

2 Comments

Have you ever wondered about the differences between Datafold's Data Diff and #dbt tests? Do data teams really need both? In this post, Elliot G. and Leo Folsom explain: ❓ Why dbt tests prevent some data quality issues, but not all 🤔 How the two tests answer fundamentally different data quality questions 🤝 And why having both in your CI pipeline is essential for complete data quality coverage https://lnkd.in/ggmC97hn

13

1 Comment

Snowflake Dynamic Iceberg Summary Tables: 1. You run near real-time Auto refresh. No need to write stored procedures and Tasks. 2. Sync with Polaris's external Catalog with RBAC features. 3. Spark can read these tables. Save Cost Snowflake

2