Jake Hannan

108 Recipies Snowflake Build: Just moments ago (Nov 12th, 12:30 PM CST), SF presented the Snowflake Dynamic Iceberg table using Polaris External CatalogSync." We have 20+ recipes in this e-book. Below is how to build summary tables as new data streaming into the base table. Snowflake

6

What's in a name? Falko Schulz creates a SAS Visual Analytics infographic on the popular names in the US collected by the Social Security Administration from 1920 to 2022. There's even a link to the interactive report if you want to delve into the data. #infographic #dataviz #SAS #SASviya #SASusers #SAStip #data https://lnkd.in/guhBVUPU

22

1 Comment

There was a whole lot of talk about GenAI at the Snowflake and Databricks Summits this year, but don't be fooled – the star of both shows wasn't actually the flashy LLM demos. It was data enablement. The announcements that made the biggest splash this year were the ones that focused on the new ways Snowflake and Databricks customers can enable data for a given use-case. The features that will help them get the right tools in place to deliver structured, governed, and reliable data into a modern data platform. So what were these announcements? We compare and contrast the events to understand the biggest news, what it says about the state of the data industry, and why data will always be the star of AI. Check it out: https://lnkd.in/eYA3kCEb #GenAI #DataCloudSummit #DataAISummit #dataquality #dataenablement

23

1 Comment

Custom date hierarchies are one of the most underutilized features in Tableau. They allow you to define specific drill-downs not automatically available. For example, you may want to allow your users to only drill-down from Year to Month.

167

4 Comments

Databricks Apps is in public preview, and it's awesome. Here are 5 things we love about it...and why 👇 𝗨𝗻𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗹𝗲𝗱 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗦𝗶𝗺𝗽𝗹𝗶𝗰𝗶𝘁𝘆 • Leverage familiar Python frameworks: Build with Dash, Shiny, Gradio, Streamlit, and Flask. • Seamless IDE integration: Develop in Visual Studio Code or PyCharm, your choice. • Pre-built templates: Jumpstart your app development with ready-to-use Python templates. • Local or cloud development: Work in your preferred environment, then deploy effortlessly. 𝗖𝘂𝘁𝘁𝗶𝗻𝗴-𝗘𝗱𝗴𝗲 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 • Serverless compute: Apps run on auto-provisioned infrastructure, eliminating manual setup. • Git integration: Benefit from version control for collaborative, trackable development. • CI/CD pipeline support: Streamline your workflow with integrated continuous integration and deployment. • Scalability: Automatically handle varying workloads without manual intervention. 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲-𝗚𝗿𝗮𝗱𝗲 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 • Unity Catalog integration: Centralized data governance across your entire Databricks ecosystem. • Granular access controls: Ensure data is accessed only by authorized personnel. • Automated service principals: Simplify and secure app-to-service authentication. • SSO and OIDC/OAuth 2.0: Robust user authentication mechanisms built-in. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 • Access to Databricks SQL: Harness the power of SQL queries directly within your apps. • AI model serving: Integrate machine learning models seamlessly into your applications. • ETL job integration: Connect your apps to data pipelines for real-time insights. • Unified data access: Leverage Delta Lake, data warehouses, and data lakes effortlessly. 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗘𝘅𝗽𝗮𝗻𝘀𝗶𝗼𝗻 𝗮𝗻𝗱 𝗩𝗮𝗹𝘂𝗲 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻 • Databricks Marketplace: Partners can now build, package, and sell data products and AI assets. • Self-service analytics: Empower business users with custom, governed data applications. • Rapid prototyping: Quickly iterate on ideas and bring concepts to life. • Cross-functional collaboration: Bridge the gap between data teams and business stakeholders. Yes. We partnered up with Databricks. But we didn't 𝗵𝗮𝘃𝗲 to. Innovations like Databricks Apps make us just 𝘄𝗮𝗻𝘁 to. 🙂

14

❄Snowflake Arctic: The Coolest Open-Source LLM on the Block?❄ Breaking New Ground: Snowflake Arctic is a cutting-edge, enterprise-focused Large Language Model (LLM) that is setting new standards in efficiency and accessibility within the AI landscape. Known for its ability to excel at complex tasks like SQL generation, coding, and instruction following, Arctic represents a leap forward in cost-effective AI training. With its unmatched capabilities, it empowers enterprises to create custom models that are not only high in quality but also economical, making advanced AI more accessible than ever before. Blueprint of Brilliance: At its core, Snowflake Arctic utilizes a pioneering architecture called the Dense-MoE Hybrid transformer. This structure combines a 10B dense transformer model with a residual configuration of 128×3.66B Mixture of Experts (MoE) MLP. This results in a robust model featuring 480 billion total parameters and 17 billion actively chosen parameters through a sophisticated top-2 gating mechanism. The training of Arctic is divided into three stages, each focusing on different data compositions: the initial phase emphasizes generic skills, while the subsequent phases concentrate on enterprise-specific abilities. This strategic curriculum is designed to fine-tune Arctic's proficiency in key business areas, enhancing its effectiveness and efficiency in real-world applications. Building the Future: The significance of Snowflake Arctic extends beyond its technical achievements. With an Apache 2.0 license, Arctic offers complete openness by providing ungated access to its weights, code, data recipes, and research insights. This level of transparency fosters a collaborative environment where the community can contribute to and benefit from the Arctic's advancements. Additionally, by performing comparably or better than established models while consuming significantly less computational resources, Arctic sets a new benchmark for sustainable AI development. It meets and exceeds the diverse needs of enterprises, driving them towards more sustainable and responsible AI usage, illustrating a future where technology and environmental considerations go hand in hand. 🔶 Taken from Yesterday's Newsletter 🔶 Link in Comments ⤵

4

Want to learn more about how to get started using Snowflake Cortex LLM Functions? Check out this post from Jon Coulter detailing how to use these tools to quickly classify data, empowering Data Engineers with Data Science tools! https://lnkd.in/gMTFT5xQ

14

I'm going to run a series of Flashback Friday's to pull out gems from our blog and continue to surface the thoughtful wisdom of one Taylor Brownlow. Here's the first of many for a thoughtful Friday reflection: "Over the last few years, many data teams have undergone a huge back-end transformation in their stacks. They have standardized how data goes from source tool to data warehouse. They have set up dbt, and with it, have created models and robust testing suites that ensure the data that’s consumed is clean and reliable. Most teams needed to take a huge step back from their typical operations to undergo this transformation. They had to agree to slower delivery times, fewer projects, and increased costs - but all in pursuit of a worthy goal. A healthy backend would mean everyone would benefit later. Everyone would have cleaner data, and they’d be able to get their insights more quickly. However, I see teams getting stuck in an endless pursuit of a perfectly efficient and performant back end. We lose sight of the fact that getting your stack in order was only step 1 of a big transformation - not the end goal. One HoD admits: “I totally feel that endless pursuit of perfection with no clear end goal - constantly changing goalposts.” Self-service feels like an extension of that same way of thinking. Teams are on the same dogged pursuit of a capability rather than asking what would be the most valuable thing. In short, it’s not about graduating from a modeling tool to a self-service tool, it’s about growing from enablers to drivers." What a punch in the gut! I'll attach the full article in the comments below if you'd like more...

12

1 Comment

😫 Another day, another thing I find annoying about Snowflake. You know what I love? When tools just work. You know what doesn't? Having to juggle Snowflake roles like I'm running a circus act. bigquery gets it right: "Oh, one of your roles can access this? Cool, go ahead!" Snowflake though? "PLEASE SPECIFY YOUR ROLE" "No, not that role" "No, the OTHER role" And don't even get me started on setting default warehouses... Listen, I get it. Role-based access control is important. But forcing users to manually switch between roles when they already HAVE the necessary permissions? That's just creating friction for the sake of it. Every time I need to: - Remember which role I need - Switch to that role - Do my thing - Probably switch back again Meanwhile, when I'm using BigQuery I'm just... getting work done. Sometimes having control over these types of thing is important, but Snowflake, your defaults are just terrible. Working with many clients on many stacks like we do at Outlier these things become pretty obvious pretty quick. Am I the only one? Who else feels this pain? 👇

34

25 Comments

Working on MLOps on Databricks? The platform provides many tools to help you get to production faster. One example is #DABs (Databricks Asset Bundles). #DABs allows you to integrate your #Databricks pipelines, models and experiments with your existing CI/CD pipeline. For example, if you are already using Jenkins, you can continue to do that by adding DABs to your CI/CD pipeline. DABs was one of the tools that convinced me about the viability of Databricks in industry use cases. At Tempered AI, we have been regularly delivering MLOps projects on #Databricks. If your team is facing a roadblock, and you need to get your pipeline upto date with standard practices, reach out to us directly, we would be happy to help. https://lnkd.in/g9_RHpxc

11

1 Comment

How do you consolidate data quickly... solo? Maybe you don't have a BI tool, maybe you do, but you need to blend data fast and don't have the Tableau chops (or license) to do so. Coefficient is Ian Gardiner's go-to. What's yours? #DataIntegration #DataConsolidation #GoogleSheets #Excel #HubSpot #GA4 #LinkedIn

12

🚀 Exciting News! Databricks Acquires Tabular to Unify Data Lakehouse Standards🚀 🌐 Databricks, the analytics and AI giant, has acquired data management company Tabular. This strategic move aims to create a common standard for data lakes and warehouses, ensuring organizations are no longer limited by format compatibility. 📈 Databricks' Explosive Growth: A Game-Changer for Data Engineers - Databricks has been on a meteoric rise, revolutionizing data engineering and machine learning. 🔑 What Does This Mean for Data Engineers? 1. Format Harmony: With Databricks leading the way, data engineers can bid farewell to format incompatibility. Delta Lake and Iceberg will converge into a unified standard, ensuring seamless interoperability between lakehouse formats. No more vendor lock-in! 🤝 2. Upskilling Opportunities: As Databricks continues to grow, data engineers who master its platform will be in high demand. Upskilling in Databricks opens doors to harnessing data's full potential, driving innovation, and unlocking new avenues for growth. WHAT DO YOU THINK ABOUT THE NEWS - share your thoughts in the comments below #DataEngineering #Databricks #LakehouseRevolution #TechNews

6

The flexibility of Tableau's interactivity separates it from every other tool. You can build this with only a couple of parameter actions. Want to learn how we built it in yesterday's Next-Level Tableau class? Comment "parameter action" and I'll reach out. See it in action → https://lnkd.in/gwE3kGCq

142

15 Comments

❄️ Excel Hack Alert: Freeze Panes to Navigate Large Datasets Easily! 📑 Working with large datasets? Keep your headers in view with the Freeze Panes feature. Go to View > Freeze Panes and choose the option that suits your needs (e.g., Freeze Top Row, Freeze First Column, or a custom selection). Scroll through your data without losing sight of your headers for easier analysis. Discover more Excel hacks with Vantage 365 Training’s courses. #Excel #ExcelTips #DataNavigation #Vantage365Training

1

A common Databricks cluster misunderstanding is people think Spot instances will get them huge cost savings instantly... the reality is a bit more complicated... When Spot nodes are revoked randomly in a Spark cluster, the impact on the job's runtime can vary wildly. We've seen runtimes jump up 2-3X, even with just 1 worker revoked. This can cause the total cost of a job to even surpass "on-demand" prices! What can you do about this? Unfortunately Spot is really a random chaos machine and nobody has control over it (not even Databricks). Hopefully on average your costs are lower, but a static on-demand and optimized cluster can get you much more stable and possibly cheaper jobs. Be kind to your clusters folks! Sync Computing #dataengineering #AI

34

4 Comments